I was assuming that sys.stdout would be referencing the same physical stream as iostreams::cout running in the same process, but this doesn't seem to be the case.
The following code, which makes a call to a C++ function with a python wrapper called "write", that writes to cout:
from cStringIO import StringIO
import sys
orig_stdout = sys.stdout
sys.stdout = stringout = StringIO()
write("cout") # wrapped C++ function that writes to cout
print "-" * 40
print "stdout"
sys.stdout = orig_stdout
print stringout.getvalue()
immediately writes "cout" to the console, then the separator "---...", and finally, as the return value of stringout.getvalue(), the string "stdout".
My intention was to capture in stringout also the string written to cout from C++.
Does anyone know what is going on, and if so, how I can capture what is written to cout in a python string?
Thanks in advance.
sys.stdout is a Python object that writes to standard output. It is not actually the standard output file handle; it wraps that file handle. Altering the object that sys.stdout points to in Python-land does not in any way affect the stdout handle or the std::cout stream object in C++.
With help from comp.lang.python and after some searching on this site:
As cdhowie pointed out, the standard output file handle has to be accessed at a lower level. In fact, its file descriptor can be obtained as sys.stdout.fileno() (which should be 1), and then os.dup and os.dup2 can be used.
I found this answer to a similar question very helpful.
What I really wanted was to capture the output in a string, not a file. The python StringIO class however doesn't have a file descriptor and cannot be used in place of an actual file, so I came up with the not fully satisfactory workaround in which a temporary file is written and subsequently read.
It cannot possibly be the same stream, as Python is written in
C, and not C++, and has no access to std::cout. Whether it
uses stdout or implements its own stream based on fd 1,
I don't know, but in any case, you'd be advised to flush between
writes using the two objects (Python and C++).
Related
I have generated a GeoTiff dataset in-memory using GDALTranslate() with a /vsimem/ filepath. I need access to the buffer for the actual GeoTiff file to put it in a stream for an external API. My understanding is that this should be possible with VSIGetMemFileBuffer(), however I can't seem to get this to return anything other than nullptr.
My code is essentially as follows:
//^^ GDALDataset* srcDataset created somewhere up here ^^
//psOptions struct has "-b 4" and "-of GTiff" settings.
const char* filep = "/vsimem/foo.tif";
GDALDataset* gtiffData = GDALTranslate(filep, srcDataset, psOptions, nullptr);
vsi_l_offset size = 0;
GByte* buf = VSIGetMemFileBuffer(filep, &size, true); //<-- returns nullptr
gtiffData seems to be a real dataset on inspection, it has all the appropriate properties (number of bands, raster size, etc). When I provide a real filesystem location to GDALTranslate() rather than the /vsimem/ path and load it up in QGIS it renders correctly too.
Looking a the source for VSIGetMemFileBuffer(), this should really only be returning nullptr if the file can't be found. This suggests i'm using it incorrectly. Does anyone know what the correct usage is?
Bonus points: Is there a better way to do this (stream the file out)?
Thanks!
I don't know anything about the C++ API. But in Python, the snippet below is what I sometimes use to get the contents of an in-mem file. In my case mainly VRT's but it shouldn't be any different for other formats.
But as said, I don't know if the VSI-api translate 1-on-1 to C++.
from osgeo import gdal
filep = "/vsimem/foo.tif"
# get the file size
stat = gdal.VSIStatL(filep, gdal.VSI_STAT_SIZE_FLAG)
# open file
vsifile = gdal.VSIFOpenL(filep, 'r')
# read entire contents
vsimem_content = gdal.VSIFReadL(1, stat.size, vsifile)
In the case of a VRT the content would be text, shown with something like print(vsimem_content.decode()). For a tiff it would of course be binary data.
I came back to this after putting in a workaround, and upon swapping things back over it seems to work fine. #mmomtchev suggested looking at the CPL_DEBUG output, which showed nothing unusual (and was silent during the actual VSIGetMemFileBuffer call).
In particular, for other reasons I had to put a GDALWarp call in between calling GDALTranslate and accessing the buffer, and it seems that this is what makes the difference. My guess is that GDALWarp is calling VSIFOpenL internally - although I can't find this in the source - and this does some kind of initialisation for VSIGetMemFileBuffer. Something to try for anyone else who encounters this.
I am familiar with Ruby and am trying to write a program in Crystal.
I have a file called special_file.txt that I want to read in my Crystal program, how do I do that?
Crystal is inspired by Ruby syntax and so you can often read and perform File operations in a similar manner. For example, Crystal has a File classclass which is an instance of the IO class containing a read method.
To read a file's contents on your filesystem you can instantiate a File object and invoke the gets_to_end method coming from the IO super class:
file = File.new("path/to/file")
content = file.gets_to_end
file.close
The gets_to_end method reads an entire IO objects data to a String variable.
You can also use a block invocation to achieve a similar result:
# Implicit close with `open`
content = File.open("path/to/file") do |file|
file.gets_to_end
end
Finally, the most idiomatic way to read the contents of an entire file would be the one line:
# Shortcut:
content = File.read("path/to/file")
I am writing a webservice in Django to handle image/video streams, but it's mostly done in an external program. For instance:
client requests for /1.jpg?size=300x200
python code parse 300x200 in django (or other WSGI app)
python calls convert (part of Imagemagick) using subprocess module, with parameter 300x200
convert reads 1.jpg from local disk, convert to size accordingly
Writing to a temp file
Django builds HttpResponse() and read the whole temp file content as body
As you can see, the whole temp file read-then-write process is inefficient. I need a generic way to handle similar external programs like this, not only convert, but others as well like cjpeg, ffmepg, etc. or even proprietary binaries.
I want to implement it in this way:
python gets the stdout fd of the convert child process
chain it to WSGI socket fd for output
I've done my homework, Google says this kind of zero-copy could be done with system call splice(). but it's not available in Python. So how to maximize performance in Python for these kind of scenario?
Call splice() using ctypes?
hack memoryview() or buffer() ?
subprocess has stdout which has readinto(), could this be utilized somehow?
How could we get fd number for any WSGI app?
I am kinda newbie to these, any suggestion is appreciated, thanks!
If the goal is to increase performance, you ought to examine the bottlenecks on a case-by-case basis, rather than taking a "one solution fits all" approach.
For the convert case, assuming the images aren't insanely large, the bottleneck there will most likely be spawning a subprocess for each request.
I'd suggest avoiding creating a subprocess and a temporary file, and do the whole thing in the Django process using PIL with something like this...
import os
from PIL import Image
from django.http import HttpResponse
IMAGE_ROOT = '/path/to/images'
# A Django view which returns a resized image
# Example parameters: image_filename='1.jpg', width=300, height=200
def resized_image_view(request, image_filename, width, height):
full_path = os.path.join(IMAGE_ROOT, image_filename)
source_image = Image.open(full_path)
resized_image = source_image.resize((width, height))
response = HttpResponse(content_type='image/jpeg')
resized_image.save(response, 'JPEG')
return response
You should be able to get results identical to ImageMagick by using the correct scaling algorithm, which, in general is ANTIALIAS for cases where the rescaled image is less than 50% of the size of the original, and BICUBIC in all other cases.
For the case of videos, if you're returning a transcoded video stream, the bottleneck will likely be either CPU-time, or network bandwidth.
I find that WSGI could actually handle an fd as an interator response
Example WSGI app:
def image_app(environ, start_response):
start_response('200 OK', [('Content-Type', 'image/jpeg'), ('Connection', 'Close')])
proc = subprocess.Popen([
'convert',
'1.jpg',
'-thumbnail', '200x150',
'-', //to stdout
], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
return proc.stdout
It wrapps the stdout as http response via a pipe
When I write to a file, using python open(filename, 'w+'), I get multiple lines of NULL written to the file in addition to the new text. Python 2.7.3
from sys import argv
script, filename, random = argv
my_file = open(filename, 'w+')
added_line = raw_input("Type what you want to add: ")
my_file.write(added_line)
print my_file.read()
my_file.close()
I am teaching myself and practicing opening and writing to files (obviously, I guess). I can get the program to run and prompt me for the new text. I also tried open(filename, 'a').
What am I missing?
Thank you.
I tried opening the file with the w option and as far as I know, if you open the file with w option you are only able to write to file, so you cannot run the read() method. Indeed, w+ enables you to read and write. (Note that 'w+' truncates the file). r+ opens the file for both reading and writing.
my_file.seek(0) return to the top of the file before reading, otherwise you'll just read an empty string.
Worked for me.
I would like to embed a text file with some data into my program.
let's call it "data.txt".
This text file is usually loaded with a function which requires the text file's file name as input and is eventually opened using a fopen() call... some something to the lines of
FILE* name = fopen("data.txt");
I can't really change this function and I would like the routine to open this same file every time it runs. I've seen people ask about embedding the file as a header but it seems that I wouldn't be able to call fopen() on a file that I embed into the header.
So my question is: is there a way to embed a text file as a callable file/variable to fopen()?
I am using VS2008.
Yes and No. The easiest way is to transform the content of the text file into an initialized array.
char data_txt[] = {
'd','a','t','a',' ','g','o','e','s',' ','h','e','r','e', //....
};
This transformation is easily done with a small perl script or even a small C program. You then compile and link the resulting module into your program.
An old trick to make this easier to manage with a Makefile is to make the script transform its data into the body of the initializer and write it to a file without the surrounding variable declaration or even the curly braces. If data.txt is transformed to data.inc, then it is used like so:
char data_txt[] = {
#include "data.inc"
};
Update
On many platforms, it is possible to append arbitrary data to the executable file itself. The trick then is to find it at run time. On platforms where this is possible, there will be file header information for the executable that indicates the length of the executable image. That can be used to compute an offset to use with fseek() after you have opened the executable file for reading. That is harder to do in a portable way, since it may not even be possible to learn the actual file name of your executable image at run time in a portable way. (Hint, argv[0] is not required to point to the actual program.)
If you cannot avoid the call to fopen(), then you can still use this trick to keep a copy of the content of data.txt, and put it back in a file at run time. You could even be clever and only write the file if it is missing....
If you can drop the call to fopen() but still need a FILE * pointing at the data, then this is likely possible if you are willing to play fast and loose with your C runtime library's implementation of stdio. In the GNU version of libc, functions like sprintf() and sscanf() are actually implemented by creating a "real enough" FILE * that can be passed to a common implementation (vfprintf() and vfscanf(), IIRC). That faked FILE is marked as buffered, and points its buffer to the users's buffer. Some magic is used to make sure the rest of stdio doesn't do anything stupid.
For any kind of file, base on RBerteig anwser you could do something simple as this with python:
This program will generate a text.txt.c file that can be compiled and linked to your code, to embed any text or binary file directly to your exe and read it directly from a variable:
import struct; # Needed to convert string to byte
f = open("text.txt","rb") # Open the file in read binary mode
s = "unsigned char text_txt_data[] = {"
b = f.read(1) # Read one byte from the stream
db = struct.unpack("b",b)[0] # Transform it to byte
h = hex(db) # Generate hexadecimal string
s = s + h; # Add it to the final code
b = f.read(1) # Read one byte from the stream
while b != "":
s = s + "," # Add a coma to separate the array
db = struct.unpack("b",b)[0] # Transform it to byte
h = hex(db) # Generate hexadecimal string
s = s + h; # Add it to the final code
b = f.read(1) # Read one byte from the stream
s = s + "};" # Close the bracktes
f.close() # Close the file
# Write the resultan code to a file that can be compiled
fw = open("text.txt.c","w");
fw.write(s);
fw.close();
Will generate something like
unsigned char text_txt_data[] = {0x52,0x61,0x6e,0x64,0x6f,0x6d,0x20,0x6e,0x75...
You can latter use your data in another c file using the variable with a code like this:
extern unsigned char text_txt_data[];
Right now I cant think of two ways to converting it to readable text. Using memory streams or converting it to a c-string.