I am wondering in which scenarios attachments would have a null value in the file_size column.
In particular I am looking at activity attachments. When is the file_size value populated? Does Siebel read the file properties at the point of upload?
I am looking at data quality and at this point I am unsure as to why a lot of attachments have no file_size value stored.
Yes, Siebel populates all the attachment information (name, extension, size, etc.) when the file is created.
Notice that there are two types of attachment: FILE and URL. You can distinguish them with the FILE_SRC_TYPE column. The field is also exposed in most attachment applets. For the URL attachments, the file size is always stored with a NULL value.
If you find any file-type attachment with null or 0 size, that's incorrect data. I can only imagine 3 scenarios where this could happen:
An error in the file creation.
A custom script updating the field value. I don't know if that's even possible, it's a system column.
Somebody updating the value directly in the database.
There is a fourth scenario: uploading an empty file, which actually has 0 size. However, in this case, the file size shown in Siebel wouldn't be 0. I've just done the test, and Siebel shows a file size of 71 bytes. According to the Bookshelf, the file size displayed in the GUI represents the size of the compressed .saf file, not the actual file size. Which also explains why URL attachments don't have a size at all.
Related
I need to read some text files that contain a huge amount of data, say 4 files each of about 500MB.
Each file contains several lines and each line has about this format:
id timestamp field1 field2 field3 field4
My strategy so far was to parse each file and for every line creating a QTreeWidgetItem with a suitable number of fields to store that line (this because during the program I want to show some of these data in a QTreeWidget) and appending all these items to a QList.
This QList is stored for all the execution of the program, in this way data are always available and I don't need to parse the files anymore.
I need all the data available because at each moment I need to access to data relative to a particular timestamp interval.
However this strategy seems too expansive in terms of resources, because I saw that the program consumes several GBs of memory and it eventually crashes.
How can I approach in a better way the handling of such data?
What you want is called 'lazy loading'.
There is an Example in the Qt documentation which shows you, how to use QAbstractItemModel, canFetchMore() and fetchMore().
From a template users can upload a csv file which gets parsed in
def parseCSV(request):
magic happens here (conforming date formats and all such fun things)
return column names to template
This view returns a list of columns and the user is asked to pick x columns to save.
The users choice is posted to
def saveCSV(request):
logic for saving
Now my question is, how do I most correctly handle the csv data object between view 1 and 2? Do i save it as a temperary file or do i send it back and forth view1->template->view2 as a data object? Or maybe something third?
There is no "correct" way as it all depends on the concrete situation. In this case, it depends on the size of the data from the CSV file. Given that the data is rather large, the best approach is most likely to store the parsed data on the server, and then in the next request only send the user's selection of the full data set.
I would suggest you to parse the data and store it as a JSON blob in the database, so that you can easily retrieve it for the next request. This way you can send the user's selection of rows and columns (or "coordinates"), and save that as real data afterwards. The benefit of storing it right away is that the user can return to the process even after leaving the flow. The downside is, though, that you save unused data, if the user never completes the process, and you might need to clear this later. If you store it in a table containing only temporary data, it should ease the cleaning process.
I would like to parse the CSV file at the frontend and give an option to user to choose columns. After choosing columns, I would send these columns with value to Backend.
After doing some research I found that the maximum size of an item (one row in a table) is 400 KB.
Source of research: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-items
I wanted to insert a text data that contains over 1 MB of size. This is basically one row worth of data.
For example,
I have a table called users which contains summary of the user.
The summary is a text field (String) and I want to insert data over 1 MB. But Dynamo DB allows only 400 KB.
Note
I cannot store this in a file and keep a pointer
You can not store more than 400 KB worth of data in one DynamoDB item (or record).
This link, that you shared in your comment, requires you to break larger records into multiple items and handle merges in your application layer. This isn't supported transparently by DynamoDB.
Just want to note that a tactical way to store items larger than 400KB into DynamoDB tables is to use compression algorithms and save the compressed data as a binary attribute.
Clients need to compress and uncompress the data.
More information here.1 and here.2
As the link mentions, you can break down the text summary into smaller chunks, let say of size w. Use indexes 1...n to store it. The value of n can be kept in another table.
What additional advantage this will give you is you can fetch the summary chunks parallelly.
I have to read data from recordset for one coloumn whose datatype is text. Below mentioned is the syntax:
VARIANT vProps = rs->Fields->GetItem("Props")->GetValue();
Here, "Props" is the name of the column which contains the xml data in text datatype. When I tried to get the value in vProps it returns some invalid value and data is not populated in variant.
I have saved the content of the column field in a .xml file and file loaded properly. Please note the size of the file is 74kb.
Likewise there are other entries also in database table for "Props" column, however in case when size is less than 10 KB, variant populated successfully using the same piece of code.
Is there any limitation to VARIANT datatypes size ?
How to fix this issue.
Thanks in advance.
Microsoft has defined limitations on the maximum sizes and numbers of various objects defined in SQL Server components.
For example,
Bytes per short string column: 8,000
I just wondering is there a way to specify returned fields for search request to the backend elasticsearch. See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html for how to specify list in JSON API.
Let me explain why i need this. I have lots of articles with a large text data. Searching in this case is very slow, cause elasticsearch returns a whole large texts for each search results, but i want to render only titles except a whole text.
May be is there another way to do it?
There are multiple options here
You can use the fields option in Elasticsearch to specify the list of fields value that has to be returned. This will save some latency time as only less data has to be transported back. But then actual data would be stored as _source and it has to be fetched from hard disk and deserialized for each call.
LINK - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html
In case we don't want to retrieve this field but you just want that field to be searchable. You can disable _source and enable store for each field whose data needs to be retrievable.
LINK , _source - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html
LINK , store - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-store.html
Djanko haystack documentation - http://django-haystack.readthedocs.org/en/latest/searchresult_api.html#SearchResult.get_additional_fields