I'm sending from server to clients base64 strings with decoded coordinates of some objects. There are ~20 objects with x;y pair, 2 bytes per integer. This coords changes with time but some of them saves same values for about 2-3 sending calls.
Any ways to calculate the difference and send it instead of full base64 each time? The network traffic is very important here.
Here is the example of 2 strings made with 100ms pause:
AFg7IP+SAAJg/ana/zAA52BJO/D/9wAxIFkAIABIABQBSADtAFEAMGlLctX/
AFo7IP+SAAJgAKnb/0EA6GBJO/D//wA0IFkAIABIABQBSADtAEoAYmlLctX/
First, pack the data efficiently in bytes, then encode if necessary. As #twall says, you should try hard to eliminate the base 64 because it expands the size of the data by 33% at least. Here is one way to pack it, if there are exactly 20 x,y pairs:
Bytes 1-3: bitset. Each bit represents an x,y pair. If set, there is an updated value for that pair in this message. Last 4 bits of 3rd byte unused.
Bytes 5&6 x coord of first point
Bytes 7&8 y coord of first point
... up to 19 more points
Max of 83 bytes, min of 3 bytes (no coords updated)
Related
I am Currently Working on Direct11 Graphics API, I am Wondering Why Index buffer has This Formula?
Start of Index Buffer = Index Buffer Base Address + Offset (bytes) + StartIndexLocation * ElementSize (bytes);
I am Wondering What Does This Mean . Someone please Help Me With This
Full Reference Docs ->
https://learn.microsoft.com/en-us/windows/win32/direct3d11/overviews-direct3d-11-resources-buffers-intro
The indices you gonna fetch depend on the following pieces of data.
The content of the index buffer.
Offset that you have passed to IASetIndexBuffer. That value is in bytes.
StartIndexLocation that you have passed when you submit a draw call such as DrawIndexed. That one is in elements, not in bytes. Direct3D 11 only supports uint16_t and uint32_t index formats, this means element size in the formula is either 2 or 4 bytes, depending on what you have passed in Format argument of IASetIndexBuffer.
After you called DrawIndexed, when GPU is doing what you asked, it combines these pieces of data to fetch a continuous range of primitives from the index buffer. The formula specifies where that range starts.
I want a high performance field of view, on a grid like a rougelike. I know I could use the field of view algorithms out there, but I since this is for a server, I think could use some more resources, since using a pre-calculated field of view would be very wasteful to use on a client/single player.
What I'm thinking of is putting all the possibilities (I'm going with 21x21 map, so there are 441 spots: 441! / (2! (441 - 2)!) = 97020) into a file, the server reads the file, and puts the data into a unordered map of a vector of pairs of ints (the coord would be based off the position of the client, e.i.; *pair*.first+client.x-11). Then when a client sends a movement, the server gets the tiles around it, hashes it using boost, and sends the visible tiles (with delta compression of course!).
I would probably mess with bits and use 9 bit bytes instead of 8 because the max size is 441, and 8 bits max is 256, and 9 bits is 510.
What I am wondering is if the hashing will take more time than using a simple algorithm (not much experience with hashing...) or if this will take too much ram, but if not then I can flat out use the most complex and accurate algorithm without the processing time!
I'm looking to filter a 1 bit per pixel image using a 3x3 filter: for each input pixel, the corresponding output pixel is set to 1 if the weighted sum of the pixels surrounding it (with weights determined by the filter) exceeds some threshold.
I was hoping that this would be more efficient than converting to 8 bpp and then filtering that, but I can't think of a good way to do it. A naive method is to keep track of nine pointers to bytes (three consecutive rows and also pointers to either side of the current byte in each row, for calculating the output for the first and last bits in these bytes) and for each input pixel compute
sum = filter[0] * (lastRowPtr & aMask > 0) + filter[1] * (lastRowPtr & bMask > 0) + ... + filter[8] * (nextRowPtr & hMask > 0),
with extra faff for bits at the edge of a byte. However, this is slow and seems really ugly. You're not gaining any parallelism from the fact that you've got eight pixels in each byte and instead are having to do tonnes of extra work masking things.
Are there any good sources for how to best do this sort of thing? A solution to this particular problem would be amazing, but I'd be happy being pointed to any examples of efficient image processing on 1bpp images in C/C++. I'd like to replace some more 8 bpp stuff with 1 bpp algorithms in future to avoid image conversions and copying, so any general resouces on this would be appreciated.
I found a number of years ago that unpacking the bits to bytes, doing the filter, then packing the bytes back to bits was faster than working with the bits directly. It seems counter-intuitive because it's 3 loops instead of 1, but the simplicity of each loop more than made up for it.
I can't guarantee that it's still the fastest; compilers and especially processors are prone to change. However simplifying each loop not only makes it easier to optimize, it makes it easier to read. That's got to be worth something.
A further advantage to unpacking to a separate buffer is that it gives you flexibility for what you do at the edges. By making the buffer 2 bytes larger than the input, you unpack starting at byte 1 then set byte 0 and n to whatever you like and the filtering loop doesn't have to worry about boundary conditions at all.
Look into separable filters. Among other things, they allow massive parallelism in the cases where they work.
For example, in your 3x3 sample-weight-and-filter case:
Sample 1x3 (horizontal) pixels into a buffer. This can be done in isolation for each pixel, so a 1024x1024 image can run 1024^2 simultaneous tasks, all of which perform 3 samples.
Sample 3x1 (vertical) pixels from the buffer. Again, this can be done on every pixel simultaneously.
Use the contents of the buffer to cull pixels from the original texture.
The advantage to this approach, mathematically, is that it cuts the number of sample operations from n^2 to 2n, although it requires a buffer of equal size to the source (if you're already performing a copy, that can be used as the buffer; you just can't modify the original source for step 2). In order to keep memory use at 2n, you can perform steps 2 and 3 together (this is a bit tricky and not entirely pleasant); if memory isn't an issue, you can spend 3n on two buffers (source, hblur, vblur).
Because each operation is working in complete isolation from an immutable source, you can perform the filter on every pixel simultaneously if you have enough cores. Or, in a more realistic scenario, you can take advantage of paging and caching to load and process a single column or row. This is convenient when working with odd strides, padding at the end of a row, etc. The second round of samples (vertical) may screw with your cache, but at the very worst, one round will be cache-friendly and you've cut processing from exponential to linear.
Now, I've yet to touch on the case of storing data in bits specifically. That does make things slightly more complicated, but not terribly much so. Assuming you can use a rolling window, something like:
d = s[x-1] + s[x] + s[x+1]
works. Interestingly, if you were to rotate the image 90 degrees during the output of step 1 (trivial, sample from (y,x) when reading), you can get away with loading at most two horizontally adjacent bytes for any sample, and only a single byte something like 75% of the time. This plays a little less friendly with cache during the read, but greatly simplifies the algorithm (enough that it may regain the loss).
Pseudo-code:
buffer source, dest, vbuf, hbuf;
for_each (y, x) // Loop over each row, then each column. Generally works better wrt paging
{
hbuf(x, y) = (source(y, x-1) + source(y, x) + source(y, x+1)) / 3 // swap x and y to spin 90 degrees
}
for_each (y, x)
{
vbuf(x, 1-y) = (hbuf(y, x-1) + hbuf(y, x) + hbuf(y, x+1)) / 3 // 1-y to reverse the 90 degree spin
}
for_each (y, x)
{
dest(x, y) = threshold(hbuf(x, y))
}
Accessing bits within the bytes (source(x, y) indicates access/sample) is relatively simple to do, but kind of a pain to write out here, so is left to the reader. The principle, particularly implemented in this fashion (with the 90 degree rotation), only requires 2 passes of n samples each, and always samples from immediately adjacent bits/bytes (never requiring you to calculate the position of the bit in the next row). All in all, it's massively faster and simpler than any alternative.
Rather than expanding the entire image to 1 bit/byte (or 8bpp, essentially, as you noted), you can simply expand the current window - read the first byte of the first row, shift and mask, then read out the three bits you need; do the same for the other two rows. Then, for the next window, you simply discard the left column and fetch one more bit from each row. The logic and code to do this right isn't as easy as simply expanding the entire image, but it'll take a lot less memory.
As a middle ground, you could just expand the three rows you're currently working on. Probably easier to code that way.
Problem: i am trying to store tile data for my map class. i had the idea of using a palette per layer, the palette would describe the data in the layer which would be an array of bytes with each byte representing a tile type.
this means 1 layer of 100 million tiles would equal ~96mb. however i overlooked how much data i could actually store in a byte and it turns out i can only store 256 tiles of course. resulting in square-root of 256 * tile-size texture sizes ( in this case 256 as tile sizes are 16) . 256*256 texture sizes are too small as each palette can only have one texture. severely limiting the tiles i can have in a layer.
i am now stuck in a bind as if i use 2 bytes ( short ) instead of 1 byte to store tile data i will double my memory usage to ~192mb per layer. and i want 4 layers at the minimum. inflating the end product to 768mb of ram used. i also can not describe the data in the data as the array offset of each byte is also a description of its location.
is there a way i could store this data more efficiently. worst case scenario will involve me saving all this to the disk and buffering to memory from the disk. but i would prefer to keep it in memory.
i guess i could come up with something smart in a few hours but i thought i would ask to see if there are any common methods i am unaware of to combat this problem.
I suggest representing your data in an array which maps to the two dimensional plane using a space filling curve such as the Hilbert curve.
Then, compress this using a combination of Huffman coding and run-length encoding. This will be particularly effective if you data is often repeated locally (i.e. there are lots of sections which are all the same tile next to each other).
Do this compression in blocks of say 256 tiles. Then, have an array of offsets that indicate how far into the compressed data certain bytes numbers are.
For example, the start of the second block (tile 256) byte might be at position 103, and the start of the third block (tile 512) might be at position 192.
Then say to access the 400th tile, you can work out this is from the second block, so decompress the second block (in this case from byte 103 to byte 191) and from this get the 400 - 256 = 144 tile. Save (cache) this decompressed data for the moment, it's likely if you're getting nearby tiles they'll also be in this decompressed block. Perhaps in your array of offsets you should also include what blocks have been recently cached, and where in the cache they are.
If you wanted to allow modifications, you'd probably have to change your data structure from one large array to a vector of vectors. Have an indicator for each vector whether it is compressed or not. When doing modifications, uncompress blocks and modify them, and recompress blocks the least recently modified blocks when memory is running out.
Or, you could just dump the whole structure to a file and memory map the file. This is much simpler but may be slower depending on the compressibility of your data and your access patterns due to additional I/O.
I have an array of point data, the values of points are represented as x co-ordinate and y co-ordinate.
These points could be in the range of 500 upto 2000 points or more.
The data represents a motion path which could range from the simple to very complex and can also have cusps in it.
Can I represent this data as one spline or a collection of splines or some other format with very tight compression.
I have tried representing them as a collection of beziers but at best I am getting a saving of 40 %.
For instance if I have an array of 500 points , that gives me 500 x and 500 y values so I have 1000 data pieces.
I around 100 quadratic beziers from this. each bezier is represented as controlx, controly, anchorx, anchory.
which gives me 100 x 4 = 400 pcs of data.
So input = 1000pcs , output = 400pcs.
I would like to further tighen this, any suggestions?
By its nature, spline is an approximation. You can reduce the number of splines you use to reach a higher compression ratio.
You can also achieve lossless compression by using some kind of encoding scheme. I am just making this up as I am typing, using the range example in previous answer (1000 for x and 400 for y),
Each point only needs 19 bits (10 for x, 9 for y). You can use 3 bytes to represent a coordinate.
Use 2 byte to represent displacement up to +/- 63.
Use 1 byte to represent short displacement up to +/- 7 for x, +/- 3 for y.
To decode the sequence properly, you would need some prefix to identify the type of encoding. Let's say we use 110 for full point, 10 for displacement and 0 for short displacement.
The bit layout will look like this,
Coordinates: 110xxxxxxxxxxxyyyyyyyyyy
Dislacement: 10xxxxxxxyyyyyyy
Short Displacement: 0xxxxyyy
Unless your sequence is totally random, you can easily achieve high compression ratio with this scheme.
Let's see how it works using a short example.
3 points: A(500, 400), B(550, 380), C(545, 381)
Let's say you were using 2 byte for each coordinate. It will take 16 bytes to encode this without compression.
To encode the sequence using the compression scheme,
A is first point so full coordinate will be used. 3 bytes.
B's displacement from A is (50, -20) and can be encoded as displacement. 2 bytes.
C's displacement from B is (-5, 1) and it fits the range of short displacement 1 byte.
So you save 10 bytes out of 16 bytes. Real compression ratio is totally depending on the data pattern. It works best on points forming a moving path. If the points are random, only 25% saving can be achieved.
If for example you use 32-bit integers for point coords and there is range limit, like x: 0..1000, y:0..400, you can pack (x, y) into a single 32-bit variable.
That way you achieve another 50% compression.
You could do a frequency analysis of the numbers you are trying to encode and use varying bit lengths to represent them, of course here I am vaguely describing Huffman coding
Firstly, only keep enough decimal points in your data that you actually need. Removing these would reduce your accuracy, but its a calculated loss. To do that, try converting your number to a string, locating the dot's position, and cutting of those many characters from the end. That could process faster than math, IMO. Lastly you can convert it back to a number.
150.234636746 -> "150.234636746" -> "150.23" -> 150.23
Secondly, try storing your data relative to the last number ("relative values"). Basically subtract the last number from this one. Then later to "decompress" it you can keep an accumulator variable and add them up.
A A A A R R
150, 200, 250 -> 150, 50, 50