Created a standard NestJS project using the command: nest new project-name. After that, I installed compression npm i --save compression and plugged it into the project.
import { NestFactory } from '#nestjs/core';
import { AppModule } from './app.module';
import * as compression from 'compression';
async function bootstrap() {
const app = await NestFactory.create(AppModule);
app.use(compression());
await app.listen(3000);
}
bootstrap();
But the compression does not work, response does not have gzip.
Response in browser
I've already tried clearing the cache, restarting the browser, nothing works.
Thanks for the help
First, install the following package:
npm i --save compression
In the bootstrap function in the main.ts file, apply the compression middleware as follow
async function bootstrap() {
const app: INestApplication = await NestFactory.create(AppModule);
...
app.use(compression({
filter: () => { return true },
threshold: 0
}));
...
await app.listen(4100);
}
Then, you will see the result
Note
if your browser is open, first close it then opens it again
Control+Shift+R or Shift + F5 = Reload your current page, ignoring cached content.
If nothing else works:
const expressApp = express();
expressApp.use(compression());
const expressAdapter = new ExpressAdapter(expressApp);
const app = await NestFactory.create(AppModule, expressAdapter);
await app.listen(3000);
Related
Our app allow our clients large file uploads. Files are stored on AWS/S3 and we use Uppy for the upload, and dockerize it to be used under a kubernetes deployment where we can up the number of instances.
It works well, but we noticed all > 5GB uploads fail. I know uppy has a plugin for AWS multipart uploads, but even when installed during the container image creation, the result is the same.
Here's our Dockerfile. Has someone ever succeeded in uploading > 5GB files to S3 via uppy? IS there anything we're missing?
FROM node:alpine AS companion
RUN yarn global add #uppy/companion#3.0.1
RUN yarn global add #uppy/aws-s3-multipart
ARG UPPY_COMPANION_DOMAIN=[...redacted..]
ARG UPPY_AWS_BUCKET=[...redacted..]
ENV COMPANION_SECRET=[...redacted..]
ENV COMPANION_PREAUTH_SECRET=[...redacted..]
ENV COMPANION_DOMAIN=${UPPY_COMPANION_DOMAIN}
ENV COMPANION_PROTOCOL="https"
ENV COMPANION_DATADIR="COMPANION_DATA"
# ENV COMPANION_HIDE_WELCOME="true"
# ENV COMPANION_HIDE_METRICS="true"
ENV COMPANION_CLIENT_ORIGINS=[...redacted..]
ENV COMPANION_AWS_KEY=[...redacted..]
ENV COMPANION_AWS_SECRET=[...redacted..]
ENV COMPANION_AWS_BUCKET=${UPPY_AWS_BUCKET}
ENV COMPANION_AWS_REGION="us-east-2"
ENV COMPANION_AWS_USE_ACCELERATE_ENDPOINT="true"
ENV COMPANION_AWS_EXPIRES="3600"
ENV COMPANION_AWS_ACL="public-read"
# We don't need to store data for just S3 uploads, but Uppy throws unless this dir exists.
RUN mkdir COMPANION_DATA
CMD ["companion"]
EXPOSE 3020
EDIT:
I made sure I had:
uppy.use(AwsS3Multipart, {
limit: 5,
companionUrl: '<our uppy url',
})
And it still doesn't work- I see all the chunks of the 9GB file sent on the network tab but as soon as it hits 100% -- uppy throws an error "cannot post" (to our S3 url) and that's it. failure.
Has anyone ever encountered this? upload goes fine till 100%, then the last chunk gets HTTP error 413, making the entire upload fail.
Thanks!
Here I'm adding some code samples from my repository that will help you to understand the flow of using the BUSBOY package to stream the data to the S3 bucket. Also, I'm adding the reference links here for you to get the package details I'm using.
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/index.html
https://www.npmjs.com/package/busboy
export const uploadStreamFile = async (req: Request, res: Response) => {
const busboy = new Busboy({ headers: req.headers });
const streamResponse = await busboyStream(busboy, req);
const uploadResponse = await s3FileUpload(streamResponse.data.buffer);
return res.send(uploadResponse);
};
const busboyStream = async (busboy: any, req: Request): Promise<any> {
return new Promise((resolve, reject) => {
try {
const fileData: any[] = [];
let fileBuffer: Buffer;
busboy.on('file', async (fieldName: any, file: any, fileName: any, encoding: any, mimetype: any) => {
// ! File is missing in the request
if (!fileName)
reject("File not found!");
let totalBytes: number = 0;
file.on('data', (chunk: any) => {
fileData.push(chunk);
// ! given code is only for logging purpose
// TODO will remove once project is live
totalBytes += chunk.length;
console.log('File [' + fieldName + '] got ' + chunk.length + ' bytes');
});
file.on('error', (err: any) => {
reject(err);
});
file.on('end', () => {
fileBuffer = Buffer.concat(fileData);
});
});
// ? Haa, finally file parsing wen't well
busboy.on('finish', () => {
const responseData: ResponseDto = {
status: true, message: "File parsing done", data: {
buffer: fileBuffer,
metaData
}
};
resolve(responseData)
console.log('Done parsing data! -> File uploaded');
});
req.pipe(busboy);
} catch (error) {
reject(error);
}
});
}
const s3FileUpload = async (fileData: any): Promise<ResponseDto> {
try {
const params: any = {
Bucket: <BUCKET_NAME>,
Key: <path>,
Body: fileData,
ContentType: <content_type>,
ServerSideEncryption: "AES256",
};
const command = new PutObjectCommand(params);
const uploadResponse: any = await this.S3.send(command);
return { status: true, message: "File uploaded successfully", data: uploadResponse };
} catch (error) {
const responseData = { status: false, message: "Monitor connection failed, please contact tech support!", error: error.message };
return responseData;
}
}
In the AWS S3 service in a single PUT operation, you can upload a single object up to 5 GB in size.
To upload > 5GB files to S3 you need to use the multipart upload S3 API, and also the AwsS3Multipart Uppy API.
Check your upload code to understand if you are using AWSS3Multipart correctly, setting the limit properly for example, in this case a limit between 5 and 15 is recommended.
import AwsS3Multipart from '#uppy/aws-s3-multipart'
uppy.use(AwsS3Multipart, {
limit: 5,
companionUrl: 'https://uppy-companion.myapp.net/',
})
Also, check this issue on Github Uploading a large >5GB file to S3 errors out #1945
If you're getting Error: request entity too large in your Companion server logs I fixed this in my Companion express server by increasing the body-parser limit:
app.use(bodyparser.json({ limit: '21GB', type: 'application/json' }))
This is a good working example of Uppy S3 MultiPart uploads (without this limit increased): https://github.com/jhanitesh10/uppy
I'm able to upload files up to a (self-imposed) limit of 20GB using this code.
I want to publish files on ipfs but it's showing me an error.
Here is my code...
const ipfsClient = require('ipfs-http-client');
const ipfs = ipfsClient({host: 'ipfs.infura.io', port: 5001, protocol:
'https'});
function App() {
const [buffer, setBuffer] = useState();
const handleChange = (event) => {
event.preventDefault();
const file = event.target.files[0];
const reader = new window.FileReader();
reader.readAsArrayBuffer(file);
reader.onloadend = () =>{
setBuffer(reader.result);
}
}
const handleSubmit = async(event) => {
event.preventDefault();
console.log('submitting...')
await ipfs.add({buffer}, (error, result) => {
console.log('ipfs results');
if(error){
console.error(error);
return;
}
});
}
I am getting this error in browser...
TypeError: ipfsClient is not a function
Should be some breaking changes. Most probably the copy of the example you have are old version. If you visit the latest readme, the new version should be initiated with:
import { create } from 'ipfs-http-client'
const client = create()
const client = create(new URL('http://127.0.0.1:5002'))
const { cid } = await client.add('Hello world!')
You can rollback to use the old version by specifiying the version no #, i.e. npm install ipfs-http-client#42.0.0. Instead of npm install ipfs-http-client which always pull the latest version (53.X now).
It's also possible to view your installed version in 'package.json' file to see the version you are using and edit with the version you need, 'delete node_modules' folder and re-run npm install. But this requires you to save, which needs a parameter -s, so to run is npm install -s ipfs-http-client
Version 42, sample code should be the one you are using 'https://github.com/ipfs/js-ipfs/tree/v42.0.0'.
Version 53(or the official 1.0 release), tells that there is a breaking change if you visit the official github site; where ipfs-http-client requires a create() and not to be used directly.
I am not familiar with ipfs but i checked the official docs and they have done the first line like this:
const { CID } = require('ipfs-http-client')
Those brackets are essential
What do {curly braces} around javascript variable name mean
I tried but didn't work. Got an error: Error when evaluating SSR module /node_modules/cross-fetch/dist/browser-ponyfill.js:
<script lang="ts">
import fetch from 'cross-fetch';
import { ApolloClient, InMemoryCache, HttpLink } from "#apollo/client";
const client = new ApolloClient({
ssrMode: true,
link: new HttpLink({ uri: '/graphql', fetch }),
uri: 'http://localhost:4000/graphql',
cache: new InMemoryCache()
});
</script>
With SvelteKit the subject of CSR vs. SSR and where data fetching should happen is a bit deeper than with other somewhat "similar" solutions. The bellow guide should help you connect some of the dots, but a couple of things need to be stated first.
To define a server side route create a file with the .js extension anywhere in the src/routes directory tree. This .js file can have all the import statements required without the JS bundles that they reference being sent to the web browser.
The #apollo/client is quite huge as it contains the react dependency. Instead, you might wanna consider importing just the #apollo/client/core even if you're setting up the Apollo Client to be used only on the server side, as the demo bellow shows. The #apollo/client is not an ESM package. Notice how it's imported bellow in order for the project to build with the node adapter successfully.
Try going though the following steps.
Create a new SvelteKit app and choose the 'SvelteKit demo app' in the first step of the SvelteKit setup wizard. Answer the "Use TypeScript?" question with N as well as all of the questions afterwards.
npm init svelte#next demo-app
cd demo-app
Modify the package.json accordingly. Optionally check for all packages updates with npx npm-check-updates -u
{
"name": "demo-app",
"version": "0.0.1",
"scripts": {
"dev": "svelte-kit dev",
"build": "svelte-kit build --verbose",
"preview": "svelte-kit preview"
},
"devDependencies": {
"#apollo/client": "^3.3.15",
"#sveltejs/adapter-node": "next",
"#sveltejs/kit": "next",
"graphql": "^15.5.0",
"node-fetch": "^2.6.1",
"svelte": "^3.37.0"
},
"type": "module",
"dependencies": {
"#fontsource/fira-mono": "^4.2.2",
"#lukeed/uuid": "^2.0.0",
"cookie": "^0.4.1"
}
}
Modify the svelte.config.js accordingly.
import node from '#sveltejs/adapter-node';
export default {
kit: {
// By default, `npm run build` will create a standard Node app.
// You can create optimized builds for different platforms by
// specifying a different adapter
adapter: node(),
// hydrate the <div id="svelte"> element in src/app.html
target: '#svelte'
}
};
Create the src/lib/Client.js file with the following contents. This is the Apollo Client setup file.
import fetch from 'node-fetch';
import { ApolloClient, HttpLink } from '#apollo/client/core/core.cjs.js';
import { InMemoryCache } from '#apollo/client/cache/cache.cjs.js';
class Client {
constructor() {
if (Client._instance) {
return Client._instance
}
Client._instance = this;
this.client = this.setupClient();
}
setupClient() {
const link = new HttpLink({
uri: 'http://localhost:4000/graphql',
fetch
});
const client = new ApolloClient({
link,
cache: new InMemoryCache()
});
return client;
}
}
export const client = (new Client()).client;
Create the src/routes/qry/test.js with the following contents. This is the server side route. In case the graphql schema doesn't have the double function specify different query, input(s) and output.
import { client } from '$lib/Client.js';
import { gql } from '#apollo/client/core/core.cjs.js';
export const post = async request => {
const { num } = request.body;
try {
const query = gql`
query Doubled($x: Int) {
double(number: $x)
}
`;
const result = await client.query({
query,
variables: { x: num }
});
return {
status: 200,
body: {
nodes: result.data.double
}
}
} catch (err) {
return {
status: 500,
error: 'Error retrieving data'
}
}
}
Add the following to the load function of routes/todos/index.svelte file within <script context="module">...</script> tag.
try {
const res = await fetch('/qry/test', {
method: 'POST',
credentials: 'same-origin',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
num: 19
})
});
const data = await res.json();
console.log(data);
} catch (err) {
console.error(err);
}
Finally execute npm install and npm run dev commands. Load the site in your web browser and see the server side route being queried from the client whenever you hover over the TODOS link on the navbar. In the console's network tab notice how much quicker is the response from the test route on every second and subsequent request thanks to the Apollo client instance being a singleton.
Two things to have in mind when using phaleth solution above: caching and authenticated requests.
Since the client is used in the endpoint /qry/test.js, the singleton pattern with the caching behavior makes your server stateful. So if A then B make the same query B could end up seeing some of A data.
Same problem if you need authorization headers in your query. You would need to set this up in the setupClient method like so
setupClient(sometoken) {
...
const authLink = setContext((_, { headers }) => {
return {
headers: {
...headers,
authorization: `Bearer ${sometoken}`
}
};
});
const client = new ApolloClient({
credentials: 'include',
link: authLink.concat(link),
cache: new InMemoryCache()
});
}
But then with the singleton pattern this becomes problematic if you have multiple users.
To keep your server stateless, a work around is to avoid the singleton pattern and create a new Client(sometoken) in the endpoint.
This is not an optimal solution: it recreates the client on each request and basically just erases the cache. But this solves the caching and authorization concerns when you have multiple users.
I have been looking around option to create data seeders to add dummy data in my loopback 4 application. However I am not able to find any option in official documentation.
I have found couple of post but those refer to loopback 3, like:
Loopback: Creating a Seed Script
loopback-seed
Please point me out to documentation to do so.
EDIT:
As per suggestion I have created start.js file in scripts folder:
require('babel-register')({
presets: ['es2015']
})
module.exports = require('./seed.js')
And I have copied the script converting it to JavaScript mentioned in seed.js file. When I am running the script, I am getting error:
Cannot find module Models and Repositories
though I have typed correct path.
Actually, I'm doing it with Loopback directly like this (this is typescript):
import * as users from './users.json';
import * as Promise from 'bluebird';
import {Entity, DefaultCrudRepository} from '#loopback/repository';
import {MyApplication} from '../src/application';
import {User} from '../src/models';
import {UserRepository} from '../src/repositories';
const app = new MyApplication();
async function loadByModel<T extends Entity, ID>(items: T[], repository$: DefaultCrudRepository<T,ID>, type: { new(it: Partial<T>): T ;}){
console.log(type.toString());
let repository = await repository$;
await repository.deleteAll();
await Promise.map(items, async (item: T) => {
try{
return await repository.create((new type(item)));
} catch(e){
console.log(item);
}
}, {concurrency: 50});
}
async function load(){
await loadByModel(users, await app.getRepository(UserRepository), User);
}
app.boot().then(async () => {
await load();
console.log('done');
});
We used a separate library db-migrate to keep our migration and seed scripts out of our loopback codebase. Moreso, because db.migrate and db.update methods of juggler are not 100% accurate as mentioned in docs as well. LB4 Database Migrations
I'm having a hard time saving an image that is being picked from Expo (React Native).
https://docs.expo.io/versions/latest/sdk/imagepicker.html
It seems that React Native does not have support for uploading the selected image as blob, but does have a base64 option.
The code:
_pickImage = async () => {
let pickerResult = await ImagePicker.launchImageLibraryAsync({
allowsEditing: true,
base64: true,
aspect: [4, 4],
});
this._handleImagePicked(pickerResult);
};
_handleImagePicked(pickerResult) {
const uri = pickerResult.base64;
const img = new db.File({ name: 'test.jpg', data: uri, type: 'base64', mimeType: 'image/jpg' });
db.UserData.load(this.state.UserDataID).then(UserData => {
img.upload({ force: true }).then((file) => {
UserData.photo = "https://remarkable-apple-95.app.baqend.com/v1" + file.id;
alert(file.id)
return UserData.update();
},
(error) => { alert(error); }
);
});
}
When I console.log(pickerResult.base64) I get a super long string that looks like base64, but when this is run, the img.upload is throwing the error and it says "PersistentError: An unexpected persistent error occurred."
You're right. React Native has no support for binary data. Unfortunately Baqend does not support base64 file uploads yet.
As a workaround you have 2 options:
Use the React Native Fetch Blob library, which bypasses the limitations of React Native not supporting binary files by uploading and downloading the files directly via native code and gives back a reference to those. Your code could look similar to this:
ImagePicker.showImagePicker(options, async (response) => {
const upload = new db.message.UploadFile('files', 'uploadFetchBlob.jpg')
const body = 'RNFetchBlob-' + response.uri;
RNFetchBlob.fetch('PUT', 'https://{YOUR-APP-NAME}.app.baqend.com/v1' + upload.request.path, upload.request.headers, body).then((res) => {
db.File({ parent: 'files', name: 'uploadFetchBlob.jpg'}).url
})
});
Unfortunately this wont work with the expo client right now, but you'd have to eject your project and use 'native code'.
The second option would be not to use the baqend file endpoint directly, but upload your base64 string to a baqend module instead. There you can parse your base64 string and upload it to your files from within your backend module. You can find an example for this in our Guide. https://www.baqend.com/guide/topics/baqend-code/#handling-binary-data
Hope this helps