Storing ~100 JPEGs (24K-100K each) in a table in PostgreSQL as a bytea column.
The <img/> tag src attributes reference the SvelteKit endpoint:
<img src="/api/file/18.json" alt="Person"/>
Getting this error:
Invalid response from route /api/file/76.json: Uint8Array body must be accompanied by content-type: application/octet-stream header
export async function get({ params }) {
const { id } = params
const sql = `SELECT _id, name, type, data FROM files WHERE _id = $1;`
const { rows } = await db.query(sql, [id])
const file = rows[0]
return {
headers: {
'Content-disposition': `attachment; filename=${file.name}`,
'Content-type': file.type
},
body: file.data
}
}
As SvelteKit endpoints don't interact with req/res objects because they're only available on certain platforms, I can't just write out the bytea value as a stream to a response object but I'm not sure what the right approach is.
Also tried this SQL statement...
SELECT _id, name, type, encode(data, 'base64') as data FROM files WHERE _id = $1;
But it didn't help.
Any thoughts?
UPDATE: The problem may be related to a SvelteKit bug - see https://github.com/sveltejs/kit/issues/1438.
The limitation in SvelteKit was resolved on June 11th. I also needed to put the file.data in a new Uint8Array...
// routes/api/file/[filenum].ts
import type { RequestHandler } from '#sveltejs/kit'
import { query } from '../_db'
export const get: RequestHandler = async ({ params }) => {
const { filenum } = params
const { rows } = await query(`SELECT * FROM find_file($1);`, [filenum])
if (rows) {
const file = rows[0]
return {
headers: {
'Content-Disposition': `attachment; filename="${file.name}"`,
'Content-Type': file.type
},
body: new Uint8Array(file.data)
}
} else return {
status: 404
}
}
Here's source for the find_file() function in PostgreSQL:
CREATE OR REPLACE FUNCTION public.find_file(
integer,
OUT _id integer,
OUT name character varying,
OUT type character varying,
OUT data bytea)
RETURNS record
LANGUAGE 'sql'
COST 100
VOLATILE PARALLEL UNSAFE
AS $BODY$
SELECT _id, name, type, data FROM files WHERE _id = $1 LIMIT 1
$BODY$;
and the query() function...
import pg from 'pg'
const pgNativePool = new pg.native.Pool({
max: 10, // default
connectionString: <string> import.meta.env.VITE_DATABASE_URL,
ssl: {
rejectUnauthorized: false
}
})
type QueryResponse = (sql: string, params?: Array<any>) => Promise<pg.QueryResult<any>>
export const query: QueryResponse = (sql: string, params?: Array<any>) => pgNativePool.query(sql, params)
Related
I have two databases that I need to interact with in my code. I have a simple function that takes an object and writes it to my PostgreSQL database using Prisma. I've tested the function with Postman, and it works perfectly, but when I try to execute it using a Jest mock (using the singleton pattern found in the Prisma unit testing guide), it returns undefined indicating that it didn't interact with the database and create the new record. Here's my code:
/prisma/clinical-schema.prisma
generator client {
provider = "prisma-client-js"
output = "./generated/clinical"
}
datasource clinicalDatabase {
provider = "postgresql"
url = "postgresql://postgres:postgres#localhost:5432/clinical-data?schema=public"
}
model pcc_webhook_update {
id Int #id #default(autoincrement())
event_type String
organization_id Int
facility_id Int
patient_id Int
resource_id String?
webhook_date DateTime #default(now()) #clinicalDatabase.Timestamptz(6)
status pcc_webhook_update_status #default(pending)
status_changed_date DateTime? #clinicalDatabase.Timestamptz(6)
error_count Int #default(0)
##unique([organization_id, facility_id, patient_id, resource_id, event_type, status])
}
enum pcc_webhook_update_status {
pending
processing
processed
error
}
/prisma/clinical-client.ts
import { PrismaClient } from './generated/clinical';
const prismaClinical = new PrismaClient();
export default prismaClinical;
/testing/prisma-clinical-mock.ts
import { PrismaClient } from '../prisma/generated/clinical';
import { mockDeep, mockReset, DeepMockProxy } from 'jest-mock-extended';
import prisma from '../prisma/clinical-client';
jest.mock('../prisma/clinical-client', () => ({
__esModule: true,
default: mockDeep<PrismaClient>()
}));
beforeEach(() => {
mockReset(prismaClinicalMock);
});
export const prismaClinicalMock = prisma as unknown as DeepMockProxy<PrismaClient>;
Everything up to this point follows the conventions outlined by the Prisma unit testing docs. The only modification I made was to make it database specific. Below is my function and tests. The request object in handle-pcc-webhooks.ts is a sample http request object, the body of which contains the webhook data I care about.
/functions/handle-pcc-webhooks/handler.ts
import prismaClinical from '../../../prisma/clinical-client';
import { pcc_webhook_update } from '../../../prisma/generated/clinical';
import { requestObject } from './handler.types';
export const handlePccWebhook = async (request: requestObject) => {
try {
const webhook = JSON.parse(request.body);
// if the webhook doesn't include a resource id array, set it to an array with an empty string to ensure processing and avoid violating
// the multi-column unique constraint on the table
const { resourceId: resourceIds = [''] } = webhook;
let records = [];
for (const resourceId of resourceIds) {
// update an existing record if one exists in the pending state, otherwise create a new entry
const record: pcc_webhook_update = await prismaClinical.pcc_webhook_update.upsert({
where: {
organization_id_facility_id_patient_id_resource_id_event_type_status: {
organization_id: webhook.orgId,
facility_id: webhook.facId,
patient_id: webhook.patientId,
resource_id: resourceId,
event_type: webhook.eventType,
status: 'pending'
}
},
update: {
webhook_date: new Date()
},
create: {
event_type: webhook.eventType,
organization_id: webhook.orgId,
facility_id: webhook.facId,
patient_id: webhook.patientId,
resource_id: resourceId,
status: 'pending' // not needed
}
});
records.push(record);
}
return records;
} catch (error) {
console.error(error);
}
};
/functions/handle-pcc-webhooks/handler.spec.ts
import fs from 'fs';
import path from 'path';
import MockDate from 'mockdate';
import { prismaClinicalMock } from '../../../testing/prisma-clinical-mock';
import { createAllergyAddRecord } from './__mocks__/allergy';
import { requestObject } from './handler.types';
import { handlePccWebhook } from './handler';
describe('allergy.add', () => {
let requestObject: requestObject;
let allergyAddRecord: any;
beforeAll(() => {
requestObject = getRequestObject('allergy.add');
});
beforeEach(() => {
MockDate.set(new Date('1/1/2022'));
allergyAddRecord = createAllergyAddRecord(new Date());
});
afterEach(() => {
MockDate.reset();
});
test('should create an allergy.add database entry', async() => {
prismaClinicalMock.pcc_webhook_update.create.mockResolvedValue(allergyAddRecord);
// this is where I would expect handlePccWebhook to return the newly created database
// record, but instead it returns undefined. If I run the function outside of this
// unit test, with the same input value, it functions perfectly
await expect(handlePccWebhook(requestObject)).resolves.toEqual([allergyAddRecord]);
});
});
// This just builds a request object with the current webhook being tested
function getRequestObject(webhookType: string) {
// read the contents of request object file as a buffer, then convert it to JSON
const rawRequestObject = fs.readFileSync(path.resolve(__dirname, '../../sample-data/handle-pcc-webhook-request.json'));
const requestObject: requestObject = JSON.parse(rawRequestObject.toString());
// read the contents of the webhook file as a buffer, then convert it to a string
const rawWebhook = fs.readFileSync(path.resolve(__dirname, `../../sample-data/${webhookType}.json`));
const webhookString = rawWebhook.toString();
// set the body of the request object to the contents of the target webhook
requestObject.body = webhookString;
return requestObject;
}
Finally, here is the result of running the unit test:
So after banging my had against the wall for a few hours, I figured out the issue. In my handler.spec.ts file, I had the following line:
prismaClinicalMock.pcc_webhook_update.create.mockResolvedValue(allergyAddRecord);
what that does is mock the value returned for any create functions run using Prisma. The issue is that my function is using an upsert function, which I wasn't explicitly mocking, thus returning undefined. I changed the above line to
prismaClinicalMock.pcc_webhook_update.upsert.mockResolvedValue(allergyAddRecord);
and it started working.
I use react native and backend was built with Prisma and GraphQL (Apollo Server).
I don't store image data to Prisma but to aw3.
The problem is I want to upload several images at once to my app. So I make image column of Prisma Array [], not String.
But as using aw3, I can upload only one image at once. So even if I make image column as Array, I can't upload several images at once as Array using aw3.
When I searched people suggest 3 options in order to upload multiple files by aw3.
multi-thread
multi-processing
zip upload (amazon-lambda)
In my case(to upload files as Array),
which option is most advisable?
And can you teach me the way of doing that?
My backend code:
export const uploadToS3 = async (file, userId, folderName) => {
const { filename, createReadStream } = await file;
const readStream = createReadStream();
const objectName = `${folderName}/${userId}-${Date.now()}-${filename}`;
const { Location } = await new AWS.S3()
.upload({
Bucket: "chungchunonuploads",
Key: objectName,
ACL: "public-read",
Body: readStream,
})
.promise();
return Location;
};
We need to resolve multiple file upload promises with Promise.all. Let us refactor our code and split it into 2 functions.
// Assume that we have list of all files to upload
const filesToUpload = [file1, file2, file3, fileN];
export const uploadSingleFileToS3 = async (file, userId, folderName) => {
const { filename, createReadStream } = await file;
const readStream = createReadStream();
const objectName = `${folderName}/${userId}-${Date.now()}-${filename}`;
const response = await new AWS.S3().upload({
Bucket: "chungchunonuploads",
Key: objectName,
ACL: "public-read",
Body: readStream,
});
return response;
};
const uploadMultipleFilesToS3 = async (filesToUpload, userId, folderName) => {
const uploadPromises = filesToUpload.map((file) => {
return uploadSingleFileToS3(file, userId, folderName);
});
// Array containing all uploaded files data
const uploadResult = await Promise.all(uploadPromises);
// Add logic here to update the database with Prisma ORM
};
// Call uploadMultipleFilesToS3 with all required parameters
I have been trying to search through the documentations and I think I found a solution but it isn't working and I continue to get an error. What I am doing for a signedUrl is
const [url] = await googleCloud
.bucket(bucketName)
.file(filename)
.makePublic()
.getSignedUrl(options);
return { url };
What I added from what I found in the docs was makePublic() but this throws and error as it is followed by getSignedUrl. I don't have any problem if I remove makePublic() and upload the file from the signed url but the file won't be public.
Any advice on how I should proceed?
UPDATE
The error I am seeing when trying to get a signed url with makePublic is [Unhandled promise rejection: Error: googleCloud.bucket(...).file(...).makePublic(...).getSignedUrl is not a function].
Additional information in regards to the options being send with getSignedUrl function are
const options = {
version: 'v4',
action: 'write',
expires: Date.now() + 15 * 60 * 1000, // 15 minutes
contentType: 'video/quicktime',
};
UPDATE (How frontend handles signed url)
const submitReview = async () => {
const url = await getSignedUrl({
variables: { filename: "google3.mov" }
}).then(async response => {
if (response.data && response.data.getSignedUrl) {
const url = response.data.getSignedUrl.url;
const pathUrl = url.split('?');
const videoPath = pathUrl[0];
const uploadedReponse = await uploadToGoogleCloud(url);
}
});
}
const uploadToGoogleCloud = async (url) => {
const videoFile = await fetch(video.uri);
const blob = await videoFile.blob();
const response = await fetch(url, {
method: 'PUT',
body: blob
}).then(res => console.log("thres is ", res)).catch(e => console.log(e));
}
I try use pubsub in apollo server & apollo client. but subscribed data is only null.
client dependency
"#apollo/react-hooks": "^3.1.5",
"apollo-boost": "^0.4.9",
"apollo-link-ws": "^1.0.20",
"graphql": "^15.0.0",
"react": "^16.13.1",
"react-dom": "^16.13.1",
"react-router-dom": "^5.2.0",
"react-scripts": "3.4.1",
"styled-components": "^5.1.1",
"subscriptions-transport-ws": "^0.9.16",
"typescript": "~3.7.2"
server dependency
"apollo-server": "^2.14.1",
"graphql": "^15.0.0",
"merge-graphql-schemas": "^1.7.8",
"ts-node": "^8.10.2",
"tsconfig-paths": "^3.9.0",
"typescript": "^3.9.3"
// apolloClient.ts
import { ApolloClient, HttpLink, InMemoryCache, split } from 'apollo-boost'
import { WebSocketLink } from 'apollo-link-ws'
import { getMainDefinition } from 'apollo-utilities'
const wsLink = new WebSocketLink({
uri: 'ws://localhost:4000/graphql',
options: {
reconnect: true
}
})
const httpLink = new HttpLink({
uri: 'http://localhost:4000'
})
const link = split(
// split based on operation type
({ query }) => {
const definition = getMainDefinition(query);
return (
definition.kind === 'OperationDefinition' &&
definition.operation === 'subscription'
);
},
wsLink,
httpLink,
)
const cache = new InMemoryCache()
const client = new ApolloClient({
cache: cache,
link: link,
})
export default client
// subscribe.ts
const ON_PUT_UNIT = gql`
subscription onPutUnit($code: String!) {
onPutUnit(code: $code)
}
`
const onPutResult = useSubscription(
ON_PUT_UNIT,
{ variables: {
code: code,
}}
)
// in is only null!!
console.log('subscribe', onPutResult)
-server-
onPutUnit.ts
type Subscription {
onPutUnit(code: String!): Room
}
import { pubsub } from '#src/index'
const { withFilter } = require('apollo-server')
export default {
Subscription: {
onPutUnit: {
subscribe: withFilter(
() => pubsub.asyncIterator(['PUT_UNIT']),
(payload: any, variables: any) => {
// no problem in payload & variable data
return payload.code === variables.code
}
)
}
},
}
putUnit.ts
type Mutation {
putUnit(code: String!, x: Int!, y: Int!, userName: String!): Room!
}
export default {
Mutation: {
putUnit: async (_: any, args: args) => {
const { code, x, y, userName } = args
const room = findRoom(code)
console.log(room) // no problem. normal data.
pubsub.publish('PUT_UNIT', room)
return room
},
},
}
Is it some problem? subscribe event is normally reached to client when publish. but data is is only null. I can't fine the reason.
You only specified a subscribe function for onPutUnit, without specifying a resolve function. That means the field utilizes the default resolver.
The default resolver just looks for a property with the same name as the field on the parent object (the first parameter passed to the resolver) and returns that. If there is no property on the parent object with the same name as the field, then the field resolves to null. The parent object is the value the parent field resolved to. For example, if we have a query like this:
{
user {
name
}
}
whatever the resolver for user returns will be the parent value provided to the resolver for name (if user returns a Promise, it's whatever the Promise resolved to).
But what about user? It has no parent field because it's a root field. In this case, user is passed the rootValue you set when initializing the ApolloServer (or {} if you didn't).
With subscriptions, this works a bit differently because whatever value you publish is actually passed to the resolver as the root value. That means you can take advantage of the default resolver by publishing an object with a property that matches the field name:
pubsub.publish('PUT_UNIT', { onPutUnit: ... })
if you don't do that, though, you'll need to provide a resolve function that transforms the payload you published. For example, if we do:
pubsub.publish('PUT_UNIT', 'FOOBAR')
Then our resolver map needs to look something like this:
const resolvers = {
Subscription: {
onPutUnit: {
subscribe: ...,
resolve: (root) => {
console.log(root) // 'FOOBAR'
// return whatever you want onPutUnit to resolve to
}
}
},
}
I try update my list after item remove by this article
but get Invariant Violation error.
my mutation:
const deleteFn = useMutation<FeaturedPlaylistGroupDelete, FeaturedPlaylistGroupDeleteVariables>(deleteQuery, {
update: (cache, mutationResult) => {
console.log('mutationResult', mutationResult)
const data = cache.readQuery({ query: featuredPlaylistsGroupsQuery })
console.log('cache', cache)
console.log('cacheData', data)
cache.writeQuery({
query: featuredPlaylistsGroupsQuery,
data: data.filter((item) => item.id !== mutationResult.data.featuredPlaylistGroupDelete.id),
})
},
})
featuredPlaylistsGroupsQuery:
export const featuredPlaylistsGroupsQuery = gql`
query FeaturedPlaylistGroups(
$active: Boolean
$noCategory: Boolean
$dateFrom: String
$dateTo: String
$title: String
$regions: [String!]
$categories: [String!]
) {
featuredPlaylistGroups(
active: $active
noCategory: $noCategory
dateFrom: $dateFrom
dateTo: $dateTo
regions: $regions
title: $title
categories: $categories
) {
active
category {
title
}
datetime
id
region
title
}
}
`
deleteQuery:
const deleteQuery = gql`
mutation FeaturedPlaylistGroupDelete($id: String!) {
featuredPlaylistGroupDelete(id: $id) {
active
categoryId
category {
title
}
datetime
id
region
title
}
}
`
error:
Invariant Violation: Can't find field
featuredPlaylistGroups({}) on object {
...
When you use readQuery, what's returned is what would have been returned in the data part of the response for that query. This is always an object. So for a query like
query {
foo
bar
}
You get an object like
{
"foo": "FOO",
"bar": "BAR"
}
When you call readQuery using your featuredPlaylistsGroupsQuery, you'll get an object with a single property named featuredPlaylistGroups. So your code should look more like:
const cached = cache.readQuery({ query: featuredPlaylistsGroupsQuery })
const featuredPlaylistGroups = cached.featuredPlaylistGroups.filter(item => {
return item.id !== mutationResult.data.featuredPlaylistGroupDelete.id
})
const data = {
...cached,
featuredPlaylistGroups,
}
cache.writeQuery({
query: featuredPlaylistsGroupsQuery,
data: data,
})
However, this still will not work because featuredPlaylistsGroupsQuery takes a number of variables. We need those variables in order to read and write from the cache, since each combination of variable that has been queries is stored separately in the cache. So you will either need to keep track of the variables used and call readQuery/writeQuery on all used combinations, or use something like apollo-link-watched-mutation