I am trying to develop an Active Appearance model using the VOSM package (http://www.visionopen.com/downloads/open-source-software/vosm/). I'm on Ubuntu 13.10 64b and use OpenCV-2.4.8, Boost-1.55 and VOSM-0.33.
I have successfully trained the model using facial databases, and have the trained data stored in a subfolder. When I try to test the AAM fitting on some test images, there are no saved files, but the terminal output indicates the fitting process has completed properly.
Command: test_smfitting -o trained_data/ -t "AAM_BASIC" -i images/test_2/ -r "true" -s "true"
output:
detection times = 5
Average Interation Times = 0
Averaget Detection time (in ms) = 0
There are output folders created named after the .jpg files, but they are empty.
How do I solve this?
Related
I am doing a distributed training using GCP Vertex platform. The model is trained in parallel using 4 GPU's using Pytorch and HuggingFace. After training when I save the model from local container to GCP bucket it throws me the error.
Here is the code:
I launch the train.py this way:
python -m torch.distributed.launch --nproc_per_node 4 train.py
After training is complete I save model files using this. It has 3 files that needs to be saved.
trainer.save_model("model_mlm") #Saves in local directory
subprocess.call('gsutil -o GSUtil:parallel_composite_upload_threshold=0 cp -r /pythonPackage/trainer/model_mlm gs://*****/model_mlm', shell=True, stdout=subprocess.PIPE) #from local to GCP
Error:
ResumableUploadAbortException: Upload complete with 1141101995 additional bytes left in stream; this can happen if a file changes size while being uploaded
And sometimes I get this error:
ResumableUploadAbortException: 409 The object has already been created in an earlier attempt and was overwritten, possibly due to a race condition.
As per the documentation name conflict, you are trying to overwrite a file that has already been created.
So I would recommand you to change the destiny location with a unique identifier per training so you don't receive this type of error. For example, adding the timestamp in string format at the end of your bucket like:
- gs://pypl_bkt_prd_row_std_aiml_vertexai/model_mlm_vocab_exp2_50epocs/20220407150000
I would like to mention that this kind of error is retryable as mentioned in the error documentation error docs.
In my Python code I execute
train_writer = tf.summary.FileWriter(TBOARD_LOGS_DIR)
train_writer.add_graph(sess.graph)
I can see 1.6MB file created in E:\progs\tensorboard_logs (and no other file)
but then when I execute
tensorboard --logdir=E:\progs\tensorboard_logs
it loads, but says: "No graph definition files were found." when I click on Graph.
Additionally, running tensorboard --inspect --logdir=E:\progs\tensorboard_logs
displays
Found event files in:
E:\progs\tensorboard_logs
These tags are in E:\progs\tensorboard_logs:
audio -
histograms -
images -
scalars -
Event statistics for E:\progs\tensorboard_logs:
audio -
graph
first_step 0
last_step 0
max_step 0
min_step 0
num_steps 1
outoforder_steps []
histograms -
images -
scalars -
sessionlog:checkpoint -
sessionlog:start -
sessionlog:stop -
This is TF 1.01 or so, on Windows 10.
I had similar issue. The issue occurred when I specified 'logdir' folder inside single quotes instead of double quotes. Hope this may be helpful to you.
egs: tensorboard --logdir='my_graph' -> Tensorboard didn't detect the graph
tensorboard --logdir="my_graph" -> Tensorboard detected the graph
In Tensorflows dealing with graphs, there are three parts:
1) creating the graph
2) Writing the graph to event file
3) Visualizing the graph in tensorboard
Example: Creating graph in tensorflow
a = tf.constant(5, name="input_a")
b = tf.constant(3, name="input_b")
c = tf.multiply(a,b, name="mul_c")
d = tf.add(a,b, name="add_d")
e = tf.add(c,d, name="add_e")
sess = tf.Session()
sess.run(c) <--check, value should be 15
sess.run(d) <--check, value should be 8
sess.run(e) <--check, value should be 23
Writing graph in event file
writer = tf.summary.FileWriter('./tensorflow_examples', sess.graph)
It is very important to specify a directory(in this case, the directory is tensorflow_examples), where the event file will be written to.
writer = tf.summary.FileWriter('./', sess.graph) didnt work for me, because the shell command => tensorboard --logdir expects a directory name.
After executing this step, verify if event file has been created in specified directory.
Visualizing graph in Tensorboard
Open terminal(bash), under working directory type:
tensorboard --logdir='tensorflow_examples' --host=127.0.0.1
Then open a new browser in http://127.0.0.1:6006/ or http://localhost/6006 and now tensorboard shows the graph successfully.
The problem might be the parameter --logdir. make sure you have type the correct
example:
in the code:
writer = tf.summary.FileWriter('./log/', s.graph)
open powershell
cd to your work directory and type
tensorboard --logdir=log
you can also use --debug to see if there is a problem in finding the log file. if you see:
TensorBoard path_to_run is: {'C:\\Users\\example\\log': None} that means it can not find the file.
You may need to change the powershell directory to your log file. And the logdir need not the single quotation marks.(Double quotation marks or without the quotes will be both OK)
Situation:
I have a project that renders product information onto a given template (custom XML format), then renders and converts it in a custom binary LCD format (steps simplified)
Our customers now want auto-fitting text container. (customer gives a box of specific size and all kinds of strings have to get auto-resized to fit into that container
For that I have to calculate the width of the string (freetype: each char/glyph) for multiple font-sizes (e.g. 100pt doesnt fit, 99pt doesnt fit, 98pt doesnt..., ..., 65pt fits!)
Problem:
The Problem is that freetype takes a lot of time (~20-30 ms) for each auto-fit element and I have only ~100ms for my whole application to use. (so when customer adds 5 more autofit elements it's already guaranteed to exceed ~100 ms)
Attempts:
A selfmade font-cache-generator which takes a font-file and calculates the widths of each unicode-character for font-sizes from 1pt to 100pt. Then it generates C source code out of the data like this:
//
#define COUNT_SIZES 100 // Font-Size 1-100
#define COUNT_CHARS 65536 // Full Unicode Table
int char_sizes[COUNT_SIZES][COUNT_CHARS] =
{
{1,1,2,2,3,1,1,2,2,3,1,2,2,1,2,2,3,1,2,.......// 65536
{2,2,3,3,4,2,1,3,3,4,2,3,3,2,3,3,4,2,3,.......// 65536
{2,3,4,3,5,2,2,4,4,5,2,4,4,2,4,3,5,3,3,.......// 65536
// ...
// 100 font sizes
};
That compiled in a dynamic lib (.so) is 25 MB in size and takes ~50ms to "dlload" and ~10ms to "dlsym" (WAAAAAAY too much!)
The same way but only ASCII table (so only 128 of 65536) compiles into a 58 KB .so file and takes ~500µs for "dlload" and ~100µs for "dlsym" (very nice!)
My next attempt would be to integrate the font-cache-generator into my project and cache only the glyphs I need for the specific customer (customer in europe needs ~500 glyphs, one in asia (e.g. traditional chinese) needs ~2500 (only examples, not exactly sure, maybe even more needed)
But before I take on that hard-work journey (:() I wanted to ask you if you know a better way of doing it? A library/project that does just that?
I cannot believe that it's not possible, how should a browser show lorem ipsum without loading seconds otherwise? :D
Any idea on how to solve this performance issue?
Any informative link on data caching with extremly fast access to cache (somewhat <1ms)?
System Info:
Unix (Ubuntu 16.04) 64bit
x86 AND arm architectures exist!
I found one possible way using these libraries:
ICU (for unicode)
Freetype (for the Glyphs)
Harfbuzz (for layout)
Github Project:
Harfbuzz-ICU-Freetype
Loose build instructions:
Search options in CMakeLists.txt option(WITH_XX "DESCRIPT." ON/OFF)
Enable CMake options with -D: cmake -DWITH_ZLIB=ON -DWITH_Harfbuzz=ON ..
mkdir build && cd build && cmake [option [option [...]]] ..
make -j $count_of_cpu_cores && sudo make install
Google for some Harfbuzz Layout tutorials / guides
I am able to run my Python script in the background to control two servos by automatically launching the script at boot up using sudo crontab -e. I modified the script so I am now continually writing the servos current position to a horz.txt and vert.txt file and using those files to initialize the steppers to their home position so I can find home position again after a power loss. The script works fine with the horz.txt and vert.txt code that I added when I manually launch the script from the command line on the black screen using sudo python mystepper6.py, but it doesn't launch automatically at startup nor does it show up as running when I type ps ax on the command line. I added a little extra code just to wiggle the servos before the main program starts and and the servos automatically wiggle as programmed in the sudo crontab -e but then it just stops and won't continue to find the home position. It seems to be something with the new code but I don't know what it could be. My sudo crontab -e line is #reboot (sleep2;python /home/pi/mystepper6.py) &. Below is the script for mystepper6.py.
The script below was shortened to show the relevant lines due to limited space available in this post. The sudo reboot hangs with no error message near the bottom where it is commented 'move to home position' (it doesn't move). I am able to print adjustv and adjusth integers prior to that point so the variables are properly assigned a value from the .txt files at that point. When I manually run mystepper6.py from the command line it runs perfectly.
I have spent waaay too many hours on this problem and have purchased two e-books to no avail. Please help.
import RPi.GPIO as gpio # import library RPi.GPIO gpio=use general purpose input output pin names
import time # import time library
PINSh = [27,10,18,23] # variable 'PINS' holds a list of gpio pin numbers
SEQAh = [(27,),(10,),(18,),(23,)]
PINSv = [4,17,22,24] # variable 'PINS' holds a list of gpio pin numbers
SEQAv = [(4,),(17,),(22,),(24,)]
DELAY = 0.01 # time between motor steps (too small of a number then the motor stalls)
alpha = 140 # horizontal full scale viewing angle in motor counts 128 counts = 360 degrees
beta = 30 # was 15 vertical full scale viewing angle in motor counts 128 counts = 360 degrees
gpio.setmode(gpio.BCM) # tells RPi.GPIO we want to use pin names (BCM is the mfg)
for pin in PINSh: # pin is a variable name assigned a new value each loop; PINS is a list
gpio.setup(pin, gpio.OUT) # this says we want to use 'pin' as an output
for pin in PINSv: # pin is a variable name assigned a new value each loop; PINS is a list
gpio.setup(pin, gpio.OUT)
def stepper(sequence, pins): # def says 'stepper' is the function name (like a variable), then parameters are inside ()
for step in sequence:
for pin in pins:
if pin in step:
gpio.output(pin, gpio.HIGH)
else:
gpio.output(pin, gpio.LOW)
webcam_horz_home = open("horz.txt", "a")
webcam_horz_home.close()
webcam_vert_home = open("vert.txt", "a")
webcam_vert_home.close()
# load last position prior to power down
webcam_horz_home = open("horz.txt", "r")
rows = webcam_horz_home.readlines();
for row in rows:
adjusth = int(row)
webcam_horz_home.close()
webcam_vert_home = open("vert.txt", "r")
rows = webcam_vert_home.readlines();
for row in rows:
adjustv = int(row)
webcam_vert_home.close()
counter = 0 # move to home position
while counter < adjustv:
stepper(SEQAv, PINSv)
counter = counter + 1
# CONTINUAL PAN AND TILT OPERATION (box pattern):
I just tested this on my Pi running Raspbian and it worked.
First, I created a python script called some_script.py in the primary user's home directory.
Script contents:
import sys
import time
with open("/home/username/some_script.py.out", "w") as f:
f.write(str(time.time()) + " " + "some_script.py RAN!\n")
Then, in the terminal, I set the executable bit on the file which allows it be executed:
sudo chmod +x some_script.py
Then, also in the terminal, I typed sudo crontab -e and added the following line at the bottom of the root user's crontab:
#reboot sleep 2; /usr/bin/python /home/username/some_script.py
I then rebooted, cded to /home/username/ and confirmed that python had run at reboot and written to a file:
cat some_script.py.out
1458062009.53 some_script.py RAN!
UPDATE / SOLUTION
After confirming that the OP was able to successfully replicate the steps above, I was fairly confident that this was not an issue stemming from varying implementations of cron on different *nix systems or some other edge case. As expected, he could run python via cron, he could use the #reboot feature and he could write to disk without any sort of permission issues.
However, it still didn't work. What did solve the issue was simply to use absolute paths rather than relative paths.
Changing
webcam_horz_home = open("horz.txt", "a")
to
webcam_horz_home = open("/home/pi/horz.txt", "a")
and
webcam_vert_home = open("vert.txt", "a")
to
webcam_vert_home = open("/home/pi/vert.txt", "a")
got OP's steppers purring again :)
EXPLANATION / BACKGROUND
When executing scripts or commands via the crontab of other users (in this case root), paths relative to the regular user's home directory (eg. ~/.bashrc) will no longer be valid or will point to a different file with the same filename if it exists. The same "issue" is sometimes seen when people run servers that call scripting languages such as PHP (often executed by the www-data user).
On a Raspberry Pi, these things can cause a bit of a catch-22 when working with GPIOs because the GPIO device is locked down and not accessible to unprivileged users without a change of permissions and groups.
If OP ran his script as root by adding the cron entry via sudo crontab -e, he would have access to the GPIO device (since root has access to almost everything). However, the relative paths in his python script that implicitly reference the home folder of the non-privileged user pi would no longer be valid (since home now means the folder /root which is the home folder of the root user).
If OP ran his scripts via the non-privileged pi user's crontab by adding the cron entries via crontab -e (without sudo), the paths would be valid (home or ~/ would now mean /home/pi/) but then he wouldn't have access to the GPIO device since pi is pretty unprivileged. Note: the #reboot feature is not available in all cron implementations (some embedded and stripped down *nix distros don't have it) and often it doesn't work for users other than root.
This means that OP had two options (there might be more; i'm no Linux guru):
place the files vert.txt and horz.txt in the home folder of the root user.
simply change the paths specified in the python script and keep running the script #reboot in root's crontab.
No. 2 is probably better since it keeps the home folder of root empty and keeps OP's files in pi's home folder. root should generally not be messed with unless there's no other options such as using sudo.
I am running libsvm through weka. Its output accuracy looks good to me, so I am planning to write a svm model by myself. However, weka didn't generate any training parameter, such as number of support vector. Therefore i cannot do anything. Searching the web, i found somebody said it would generate some parameters like the following:
optimization finished, #iter = 27
nu = 0.058475864943863545
obj = -1.871013102744184, rho = -0.19357337828800944
nSV = 9, nBSV = 0 `enter code here`
Total nSV = 9
but how come i didn't see any of them? any step that i missed? please help me. Thanks a lot.
Weka writes the output you mentioned to stderr.
So if you have started weka.sh or weka.bat from a terminal (or "command window" if you are on Windows), you should see that output appear in your terminal window after clicking "classify"
If you want to have access to this information via scripts, you can
redirect the output to a file and read in that file.
Here is how to edit the startup file weka.sh / weka.bat.
Edit this line (it is probably the last line) in order to write log info to a file instead of the terminal window:
java -cp $CP -Xmx8092m weka.gui.GUIChooser 2>>/opt/weka-stable/weka.log &
You can also add a properties file to your home directory to add more fine-grained behaviour.
https://weka.wikispaces.com/Properties+file
(You probably can also access information via the Weka Java API somehow, but you did not ask for that)