I am using the memory_profiler for python as follows:
from memory_profiler import profile
#profile
def my_func():
...
And run it normally.
Output looks like this (taken from docs):
Line # Mem usage Increment Line Contents
==============================================
3 #profile
4 5.97 MB 0.00 MB def my_func():
5 13.61 MB 7.64 MB a = [1] * (10 ** 6)
6 166.20 MB 152.59 MB b = [2] * (2 * 10 ** 7)
7 13.61 MB -152.59 MB del b
8 13.61 MB 0.00 MB return a
Though I'd like it to be shortened to function level and look akin to this:
Line # Mem usage Increment Max Line Contents
==============================================================
4 13.61 MB 7.64 MB 166.20 MB my_func()
20 23.26 MB 9.65 MB 56.20 MB my_second_func()
Is this possible?
You can always define a third function
#profile
def my_third_func():
my_func()
my_second_func()
and profile this function. This should yield the output you are looking for.
Otherwise, if you are only interested in the maximum memory consumption of the different functions, mprof provides a handy visual plot for that.
Related
I have a process for cleaning files and then saving them into correctly formatted files that arrow can read at a later time. The files are tsv format and have around 30 columns, mixed data types -- mostly character, but a couple of numeric columns. There is a significant number of files that have only a header and no data content. I decided that prior to reading in the files for cleaning, I would check to ensure that they have content, rather than reading in the file as a data frame, and then checking for content. So, essentially, I just wanted to check that the number of lines in the file was >= 2. I am using a simple C++ method that I pulled into R using Rcpp:
Rcpp::cppFunction(
"
#include <fstream>
bool more_than_one_line(std::string filepath) {
std::ifstream input_file;
input_file.open(filepath);
std::string unused;
int numLines = 0;
while(std::getline(input_file, unused)) {
++numLines;
if (numLines >= 2) {
return true;
}
}
return false;
}
"
)
I take some timing measurements like so:
v <- vector(mode="numeric", length=1000)
ii = 0
for (file in listOfFiles[1:1000]) {
print(ii)
ii = ii + 1
t0 <- Sys.time()
more_than_one_line(file);
v[ii] <- difftime(Sys.time(), t0)
}
When I run this code, it takes about 1 second per file, if the files have never been read before; it's much, much faster if I run the code over files that have previously been processed. Yet, according to this SO answer, the fastest time for counting the lines in a 12M row file is 0.1 seconds (my files are max 500k rows), and the SO user who recommended that fastest strategy (which used Linux wc) also recommended that using C++ would be quite fast. I thought the C++ method I wrote would be equally as fast as the wc method, if not faster, at least due to the fact that I am only reading, at most, the first two lines.
Am I thinking about this wrong? Is my approach wrong?
In my answer to the question linked to by the OP I mention package fpeek, function peek_count_lines. This is a fast function coded in C++. With a directory of 82 CSV files ranging from 4.8K lines to 108K lines on my computer (1 year old 11th Gen Intel(R) Core(TM) i5-1135G7 # 2.40GHz running Windows 11) it gives me an average 0.03 secs per file to get the number of lines.
Then you can use these values and subset on the condition flsize >= 2.
#path <- "my/path/omited"
fls <- list.files(path, full.names = TRUE)
# how many files
length(fls)
#> [1] 82
# range of file sizes in MB
range(file.size(fls)) / 1024L / 1024L
#> [1] 0.766923 19.999812
# total file size in MB
sum(file.size(fls)) / 1024L / 1024L
#> [1] 485.6675
# this is the main problem
t0 <- system.time(
flsize <- sapply(fls, fpeek::peek_count_lines)
)
# the files have from 4.8K to 108K lines
range(flsize)
#> [1] 4882 108503
# how many files have more than just the header line
sum(flsize >= 2)
#> [1] 82
# timings
t0
#> user system elapsed
#> 0.28 1.12 2.30
# average timings per file
t0/length(flsize)
#> user system elapsed
#> 0.003414634 0.013658537 0.028048780
Created on 2023-02-04 with reprex v2.0.2
I am new to profiling. I am trying to profile my PHP with xdebug.
The cachegrind file is created but has no significant content
I have set xdebug.profiler_enable_trigger = 1
xdebug.profiler_output_name = cachegrind+%p+%H+%R.cg
I call my page with additional GET parameter ?XDEBUG_PROFILE=1
My cachegrind file is generated but has no significant content
Here is my output:
version: 1
creator: xdebug 2.7.0alpha1 (PHP 7.0.30-dev)
cmd: C:\WPNserver\www\DMResources\Classes\VendorClasses\PHPMySQLiDatabase\MysqliDb.php
part: 1
positions: line
events: Time Memory
fl=(1)
fn=(221) php::mysqli->close
1244 103 -14832
fl=(42)
fn=(222) MysqliDbExt->__destruct
1239 56 0
cfl=(1)
cfn=(221)
calls=1 0 0
1244 103 -14832
That's it - I must be missing something fundamental.
I think you hit this bug in xdebug.
As suggested by Derick in the issue tracker, you can workaround this by adding %r to the profiler output name. eg: xdebug.profiler_output_name = cachegrind+%p+%H+%R+%r.cg
(with %r adding a random number to the name)
I am trying to extract data from a k6 output (https://docs.k6.io/docs/results-output):
data_received.........: 246 kB 21 kB/s
data_sent.............: 174 kB 15 kB/s
http_req_blocked......: avg=26.24ms min=0s med=13.5ms max=145.27ms p(90)=61.04ms p(95)=70.04ms
http_req_connecting...: avg=23.96ms min=0s med=12ms max=145.27ms p(90)=57.03ms p(95)=66.04ms
http_req_duration.....: avg=197.41ms min=70.32ms med=91.56ms max=619.44ms p(90)=288.2ms p(95)=326.23ms
http_req_receiving....: avg=141.82µs min=0s med=0s max=1ms p(90)=1ms p(95)=1ms
http_req_sending......: avg=8.15ms min=0s med=0s max=334.23ms p(90)=1ms p(95)=1ms
http_req_waiting......: avg=189.12ms min=70.04ms med=91.06ms max=343.42ms p(90)=282.2ms p(95)=309.22ms
http_reqs.............: 190 16.054553/s
iterations............: 5 0.422488/s
vus...................: 200 min=200 max=200
vus_max...............: 200 min=200 max=200
The data comes in the above format and I am trying to find a way to get each line in the above along with the values only. As an example:
http_req_duration: 197.41ms, 70.32ms,91.56ms, 619.44ms, 288.2ms, 326.23ms
I have to do this for ~50-100 files and want to find a RegEx or similar quicker way to do it, without writing too much code. Is it possible?
Here's a simple Python solution:
import re
FIELD = re.compile(r"(\w+)\.*:(.*)", re.DOTALL) # split the line to name:value
VALUES = re.compile(r"(?<==).*?(?=\s|$)") # match individual values from http_req_* fields
# open the input file `k6_input.log` for reading, and k6_parsed.log` for parsing
with open("k6_input.log", "r") as f_in, open("k6_parsed.log", "w") as f_out:
for line in f_in: # read the input file line by line
field = FIELD.match(line) # first match all <field_name>...:<values> fields
if field:
name = field.group(1) # get the field name from the first capture group
f_out.write(name + ": ") # write the field name to the output file
value = field.group(2) # get the field value from the second capture group
if name[:9] == "http_req_": # parse out only http_req_* fields
f_out.write(", ".join(VALUES.findall(value)) + "\n") # extract the values
else: # verbatim copy of other fields
f_out.write(value)
else: # encountered unrecognizable field, just copy the line
f_out.write(line)
For a file with contents as above you'll get a resulting:
data_received: 246 kB 21 kB/s
data_sent: 174 kB 15 kB/s
http_req_blocked: 26.24ms, 0s, 13.5ms, 145.27ms, 61.04ms, 70.04ms
http_req_connecting: 23.96ms, 0s, 12ms, 145.27ms, 57.03ms, 66.04ms
http_req_duration: 197.41ms, 70.32ms, 91.56ms, 619.44ms, 288.2ms, 326.23ms
http_req_receiving: 141.82µs, 0s, 0s, 1ms, 1ms, 1ms
http_req_sending: 8.15ms, 0s, 0s, 334.23ms, 1ms, 1ms
http_req_waiting: 189.12ms, 70.04ms, 91.06ms, 343.42ms, 282.2ms, 309.22ms
http_reqs: 190 16.054553/s
iterations: 5 0.422488/s
vus: 200 min=200 max=200
vus_max: 200 min=200 max=200
If you have to run it over many files, I'd suggest you to investigate os.glob(), os.walk() or os.listdir() to list all the files you need and then loop over them and execute the above, thus further automating the process.
I've been doing dome tests using aerospike and I noticed a behavior different than what is sold.
I have a cluster of 4 nodes running on AWS in the same AZ, the instances are t2micro (1cpu, 1gb RAM, 25gb SSD) using the aws linux with the AMI aerospike
aerospike.conf:
heartbeat {
mode mesh
port 3002
mesh-seed-address-port XXX.XX.XXX.164 3002
mesh-seed-address-port XXX.XX.XXX.167 3002
mesh-seed-address-port XXX.XX.XXX.165 3002
#internal aws IPs
...
namespace teste2 {
replication-factor 2
memory-size 650M
default-ttl 365d
storage-engine device {
file /opt/aerospike/data/bar.dat
filesize 22G
data-in-memory false
}
}
What I did was a test to see if I would loose documents when a node goes down. For that I wrote a little code on python:
from __future__ import print_function
import aerospike
import pandas as pd
import numpy as np
import time
import sys
config = {
'hosts': [ ('XX.XX.XX.XX', 3000),('XX.XX.XX.XX',3000),
('XX.XX.XX.XX',3000), ('XX.XX.XX.XX',3000)]
} # external aws ips
client = aerospike.client(config).connect()
for i in range(1,10000):
key = ('teste2', 'setTest3', ''.join(('p',str(i))))
try:
client.put(key, {'id11': i})
print(i)
except Exception as e:
print("error: {0}".format(e), file=sys.stderr)
time.sleep(1)
I used this code just for inserting a sequence of integers that I could check after that. I ran that code and after a few seconds I stopped the aerospike service at one node for 10 seconds, using sudo service aerospike stop and sudo service aerospike colstart to restart.
I waited for a few seconds until the nodes did all the migration and executed the following python script:
query = client.query('teste2', 'setTest3')
query.select('id11')
te = []
def save_result((key, metadata, record)):
te.append(record)
query.foreach(save_result)
d = pd.DataFrame(te)
d2 = d.sort(columns='id11')
te2 = np.array(d2.id11)
for i in range(0,len(te2)):
if i > 0:
if (te2[i] != (te2[i-1]+1) ):
print('no %d'% int(te2[i-1]+1))
print(te2)
And got as response:
no 3
no 6
no 8
no 11
no 13
no 17
no 20
no 22
no 24
no 26
no 30
no 34
no 39
no 41
no 48
no 53
[ 1 2 5 7 10 12 16 19 21 23 25 27 28 29 33 35 36 37 38 40 43 44 45 46 47 51 52 54]
Is my cluster configured wrong or this is normal?
ps: I tried to include as many things I could, if you please suggest more information to include I will appreciate.
Actually I found a solution, and it is pretty simple and foolish to be honest.
In the configuration file we have some parameters for network communication between nodes, such as:
interval 150 # Number of milliseconds between heartbeats
timeout 10 # Number of heartbeat intervals to wait
# before timing out a node
This two parameters set the time it takes to the cluster to realize the node is down and out of the cluster. (in this case 1.5 sec).
What we found useful was to tune the write policies at the client to work along this parameters.
Depending on the client you will have some policies like number of tries until the operation fails, timeout for the operation, time between tries.
You just need to adapt the client parameters. For example: set the number of retries to 4 (each is executed after 500 ms) and the timeout to 2 sec. Doing that the client will recognize the node is down and redirect the operation to another node.
This setup can be overwhelming on the cluster, generating a huge overload, but it worked for us.
I am creating a GUI using tkinter.
In the code below, I am using two methods to place labels on a GUI's screen.
In the method 1, I have initialised label, and then used label_place and label_forget to place the labels on different screen.
In the second method, I use pointers to label text variable option, and I used the set() property to set the text at different positions.
I wish to know which method is efficient.
As with method 1 on every page I have to use the forget method, and then replace the labels.
In the second method, I do pointer[i].set("text") to place test, and pointer[i].set(" ") for no text .
#! /usr/bin/python
from Tkinter import *
class Gui(Frame):
def __init__(self,parent):
Frame.__init__(self,parent)
self.parent=parent
self.data_Frame()
Label_Tuple=["a:","b:","b:","d:","a:","b:","b:","d:","a:","b:","b:","d:"]
pos=[0,1,2,3,4,5,6,7,8,9,10,11]
self.label_init()
self.label_place(Label_Tuple,pos)
#self.label_forget()
#self.Data_Frame()
#self.widgets_place(Label_Tuple)
#self.widgets_forget()
def data_Frame(self):
self.data_frame=Frame(self.parent, bg = "cyan")
self.data_frame.place(x=10,y=55,anchor=NW,height=370,width=510)
self.data_frame.config(highlightbackground='black',highlightthickness='0')
def label_init(self):
self.label=[]
for i in range(0,12):
self.label.append(Label(self.data_frame,width=14,bg="cyan",anchor=NE,font=("Helvetica", 15)))
def label_place(self,Label,position):
self.place_x=13
self.place_y=5
self.incr=30
j=0
for i,label in enumerate(Label):
while True:
if position[i]==j:
self.label[i].config(text=Label[i])
self.label[i].place(x=self.place_x,y=self.place_y)
break
else:
self.place_y +=self.incr
j +=1
def label_forget(self):
for i in range(0,12):
self.para_label[i].place_forget()
def Data_Frame(self):
self.Data_Frame=Frame(self.parent, bg = "cyan")
self.Data_Frame.place(x=10,y=55,anchor=NW,height=370,width=510)
self.Data_Frame.config(highlightbackground='black',highlightthickness='0')
self.label=[]
place_x=13
place_y=10
incr=30
self.string_var0=[]
for i in range(0,12):
variable=StringVar()
self.label.append(Label(self.Data_Frame,width=14,textvariable=variable,bg="red",anchor=NW,font=("Helvetica", 15)))
self.string_var0.append(variable)
self.label[i].place(x=place_x,y=place_y)
place_y +=incr
def widgets_place(self,Label_Tuple):
for i in range(0,12):
self.string_var0[i].set(Label_Tuple[i])
def widgets_forget(self):
Label_Tuple=["","","","","","","","","","","",""]
for i in range(0,12):
self.string_var0[i].set("")
if __name__== "__main__":
root =Tk()
#root.attributes('-fullscreen',True)
root.geometry("750x490")
#root.config(cursor="none")
#bigfont = tkFont.Font(family="Helvetica",size=15,slant="roman")
#root.option_add("*TCombobox*Listbox*Font", bigfont)
#root.option_add("*TCombobox*Listbox*Font", bigfont)
app=Gui(root)
app.mainloop()
How can i check the execution speed and the memory occupied by a this python code or the code execution time taken .
I use the OS's time command. Note that it gives you all three, system time, user time, and wall clock time.
/usr/bin/time -v program_name.py
You can also use top or htop to view memory usage, but usage will always be changing when the program is running. There is also ps and smaps but I have not used either.
results are something like the following (note that modern systems
use as much of the CPU as is available and give up some of it when
necessary. I am on a multi-core system so the program had
all of one core to itself).
Command being timed: "./test_r_1.py"
User time (seconds): 0.54
System time (seconds): 0.28
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.82
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 51248
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 37484
Voluntary context switches: 57
Involuntary context switches: 739
Swaps: 0
File system inputs: 0
File system outputs: 176
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0