Shiny downloadButton() and downloadHandler() 500 Error - shiny

I have developed a Shiny Dashboard, I have several data frames that get imported through reactive file readers, etc.. I have also added a "Generate PDF" button, using downloadButton() in my ui.R code. My server.R code implements the downloadHandler() to handle that request.
On my Windows desktop this all works perfectly. I want this to run on a Linux server I have setup. I had to modify some paths, of course, and Shiny Server runs as root on this box. When I click the "Generate PDF" button on site running on the Linux server, I get an HTTP 500 error almost instantly. I have manually compiled the pdfReport.Rmd file on the Linux server myself and it runs just fine.
I am guessing one of two things:
Somehow the data isn't getting passed the same way on the Linux box as it does on the Windows desktop. This is probably not likely, but it is a possibility.
I have something wrong with my paths so when the temp files get written to start generating the PDF, the system doesn't have the ability or a path doesn't exist to write the file. Possibly my downloadHandler() code is malformed in some way. I think this is a higher possibility than the #1.
Here is my code for the downloadHandler():
output$pdfReport <- downloadHandler(
# For PDF output, change this to "report.pdf"
filename = reactive({paste0("/srv/shiny-server/itpod/","ITPOD-",Sys.Date(),".pdf")}),
content = function(file) {
# Copy the report file to a temporary directory before processing it, in
# case we don't have write permissions to the current working dir (which
# can happen when deployed).
tempReport <- file.path("/srv/shiny-server/itpod", "pdfReport.Rmd")
file.copy("report.Rmd", tempReport, overwrite = TRUE)
params <- list(ilp=updateILP(), ico=updateICO(), sec=updateSecurity(), ppwc=updateWorkPreviousPeriodCompleted(),
pow=updateOngoingWorkCABApproved(), pwcr=updatePlannedWorkCABRequested(), epca=updateEmergencyChangesPendingCABApproval(),
fac=updateFacilities(), drs=updateDRStatus(), ov=updateOperationalEvents(), sl=updateStaffLocations(),
w = updateWeather())
# Knit the document, passing in the `params` list, and eval it in a
# child of the global environment (this isolates the code in the document
# from the code in this app).
rmarkdown::render(tempReport, output_file = file, params = params, envir = new.env(parent = globalenv())
)
}
)
I thought maybe that the path just wasn't writeable, so I tried changing that to /tmp, but that didn't work either. Poking around, I discovered that when I over the "Generate PDF" button, I get a long URL with a "session":
http://my.url.com:3838/itpod/session/d661a858f5679aba26692bc9b4442872/download/pdfReport?w=
I'm starting to wonder if this is the issue and that I'm not writing to a path of the current session or something? This is a new area to me with Shiny. Like I said, on my desktop it works fine, but once I deploy it to the Linux server, it doesn't work correctly. Any help would be much appreciated. Thanks in advance!

Ok - after much troubleshooting, I figured out that some of the files I had in the shiny webroot that were dependencies for the main pdfReport.Rmd file weren't being seen, since the code copied the report to a temp directory.
Because I didn't want to copy all of the files from my webroot over to the temp, I decided to make the report render within the webroot itself. For me, this isn't a big deal since my shiny app is running as root anyway.
I will fix this now that I have it working, basically my fix will be to do the following:
Make the service run as a normal user
Rather than copy of the files that the report depends on, I will have to statically reference them in the report code.
I apologize for all of those who may have read this and are working on it. My fix was to the code above was the following:
output$pdfReport <- downloadHandler(
# For PDF output, change this to "report.pdf"
filename = reactive({paste0("/srv/shiny-server/itpod/","ITPOD-",Sys.Date(),".pdf")}),
content = function(file) {
# Copy the report file to a temporary directory before processing it, in
# case we don't have write permissions to the current working dir (which
# can happen when deployed).
report <- file.path(getwd(), "pdfReport.Rmd")
#tempReport <- file.path(tempdir(), "pdfReport.Rmd")
#file.copy("pdfReport.Rmd", tempReport, overwrite = TRUE)
params <- list(ilp=updateILP(), ico=updateICO(), sec=updateSecurity(), ppwc=updateWorkPreviousPeriodCompleted(),
pow=updateOngoingWorkCABApproved(), pwcr=updatePlannedWorkCABRequested(), epca=updateEmergencyChangesPendingCABApproval(),
fac=updateFacilities(), drs=updateDRStatus(), ov=updateOperationalEvents(), sl=updateStaffLocations(),
w = updateWeather())
# Knit the document, passing in the `params` list, and eval it in a
# child of the global environment (this isolates the code in the document
# from the code in this app).
rmarkdown::render(report, output_file = file, params = params, envir = new.env(parent = globalenv())
)
}
)
})
Notice, that instead of copying the file to a temp directory, I just specify the file in the current working directory.

Related

Do Quarto publish on quarto-pub (book format) images from web links?

I want to publish with the command
quarto::quarto_publish_site()
my book-website.
The book-website is already setup on quarto-pub. If I don't add any image as a web link, the website runs and can be uploaded.
Now I add any image as a weblink, this is a exemplary code
![](https://www.website.com/wp-content/uploads/sites/2/2022/04/picture.jpg)
When I render it locally, it works. When I launch the command to publish it
compilation failed- error Unable to load picture or PDF file 'https://www.website.com/wp-content/uploads/sites/2/2022/04/picture.jpg'.
The publishing process is interrupted after this error. This is exactly the same if I launch the command from Terminal.
Is this intended to prevent to publish on quarto-pub links from other websites?
Or I can do something to avoid to download all these pics?
Including images via URL is not supposed to work for PDF output, which is not a Quarto issue but comes from how Pandoc translates !()[] to LaTeX.
Instead, you could automatically generate a local copy of the file (if not available) and then include the image in an R code chunk like this:
```{r, echo=FALSE, fig.cap='Kid', dpi=100}
if(!file.exists("kid.jpg")) {
download.file(
url = "https://edit.co.uk/uploads/2016/12/Image-1-Alternatives-to-stock-photography-Thinkstock.jpg",
destfile = "kid.jpg",
mode = "auto")
}
knitr::include_graphics("kid.jpg")
```
(of course, including the image via !()["kid.jpg"] at different location will work too once the file exists locally.)

Right way to delay file download in Django

I have a class-based view which triggers the composition and downloading of a report for a user.
Normally in def get of the class I just compile the report, add response['Content-Disposition'] = 'attachment; filename="somefilename.pdf"' and return response to a user.
The problem is that some reports are large and while they are compiling the request timeout happens.
I know that the right way of dealing with this is to delegate it to a background process (like Celery). But the problem is that it means that instead of creating a temporary file which ceases to exist the moment the user downloads a report, I have to store these reports somewhere, and write a cronjob which will regularly clean the reports directory.
Is there any more elegant way in Django to deal with this issue?
One solution less fancy than using celery is to use is Django's StreamingHttpResponse:
(https://docs.djangoproject.com/en/2.0/ref/request-response/#django.http.StreamingHttpResponse
With this, you use a generator function, which is a python function that uses yield to return its results as an iterator. This allows you to return the data as you generate it, rather than all at once at after you're finished. You can yield after each line or section of the report.. thus keeping a flow of data back to the browser.
But.. this only works if you are building up the finished file bit by bit.. for example, a CSV file. If you're returning something that you need to format all at once, for example if you're using something like wkhtmltopdf to generate a pdf file after you're done, then it's not as easy.
But there's still a solution:
What you can do in that case is, use StreamingHttpReponse along with a generator function to generate your report into a temporary file, instead of back to the browser. But as you are doing this, yield HTML snippets back to the browser which lets the user know the progress, eg:
def get(self, request, **kwargs):
# first you need a tempfile name.. do that however you like
tempfile = "kfjkdsjfksjfks"
# then you need to create a view which will open that file and serve it
# but I won't show that here.
# For security reasons it has to serve only out of one directory
# that is dedicated to this.
fetchurl = reverse('reportgetter_url') + '?file=' + tempfile
def reportgen():
yield 'Starting report generation..<br>'
# do some stuff to generate your report into the tempfile
yield 'Doing this..<br>'
# do this
yield 'Doing that..<br>'
# do that
yield 'Finished.<br>'
# when the browser receives this script, it'll go to fetchurl where
# you will send them the finished report.
yield '<script>document.location="%s";</script>' % fetchurl
return http.StreamingHttpResponse(reportgen())
That's not a complete example obviously, but should give you the idea.
When your user fetches this view, they will see the progress of the report as it comes along. At the end, you're sending the javacript which redirect the browser to the other view you will have to write which returns the response containing the finished file. When the browser gets this javacript, if the view returning the tempfile is setting the response Content-Disposition as an attachment before returning it, eg:
response['Content-Disposition'] = 'attachment; filename="%s"' % filename
..then the browser will stay on the current page showing your progress.. and simply pop up a file save dialog for the user.
For cleanup, you'll need a cron job regardless.. because if people don't wait around, they'll never pick up the report. Sometimes things don't work out... So you could just clean up files older than let's say 1 hour. For a lot of systems this is acceptable.
But if you want to clean up right away, what you can do, if you are on unix/linux, is to use an old unix filesystem trick: Files which are deleted while they are open are not really gone until they are closed. So, open your tempfile.. then delete it. Then return your response. As soon as the response has finished sending, the space used by the file will be freed.
PS: I should add.. that if you take this second approach, you could use one view to do both jobs.. just:
if `file` in request.GET:
# file= was in the url.. they are trying to get an already generated report
with open(thepathname) as f:
os.unlink(f)
# file has been 'deleted' but f is still a valid open file
response = HttpResponse( etc etc etc)
response['Content-Disposition'] = 'attachment; filename="thereport"'
response.write(f)
return response
else:
# generate the report
# as above
This is not really a Django question but a general architecture question.
You can always increase your server time outs but it would still, IMO, give you a bad user experience if the user has to sit watching the browser just spin.
Doing this on a background task is the only way to do it right. I don’t know how large the reports are, but using email can be a good solution. The background task simply generates the report, sends it via email and deletes it.
If the files are too large to send via email, then you will have to store them. Maybe send an email with a link and a message indicating the link will not work after X days/hours. Once you have a background worker, creating a daily or hourly clean up task would be very easy.
Hope it helps

Power Query : How to handle "DataSource.NotFound: File or Folder: We couldn't find the folder " error

I'm trying to get the folder contents using the query below and also handles the error. But still I'm getting the text with a warning icon - "DataSource.NotFound: File or Folder: We couldn't find the folder"
See the M-code below:
let
Source = Folder.Files("\\serverpath\Desktop\"),
AlternativeOutput=#table(type table [Name=text,Extension=text,Availability=text], {{"Error", "Error", "Folder not available"}}),
TestForError= try Source,
Output =
if TestForError[HasError] then AlternativeOutput
else Source
in
Output
This is likely a bug, which explains why try...otherwise also doesn't work. The HasError field is returning false even though we get a DataSource.NotFound error. I've filed an item on our end to track this issue.
You need to force evaluation of the Folder.Files at each try / otherwise attempt, using Table.Buffer
= try Table.Buffer(Folder.Files(path1)) otherwise Folder.Files(path2)
you can then evaluate multiple paths, such as the same onedrive account if used on different computers having different login credentials i.e named username on one pc, but user.name on another (e.g personal computer, work computer)
= let SourceA = try Folder.Files("C:\Users\Username\OneDrive\Documents\"),
SourceB = try Folder.Files("C:\Users\User.name\OneDrive\Documents\"),
DynamicSource = try Table.Buffer(SourceA[Value])
otherwise Table.Buffer(SourceB[Value])
in DynamicSource 

How to control the download of files with Selenium + Python bindings in Chrome

Where can I find the documentation that describes the options I can use with Selenium and Chrome web browser? I want to open a link in a web browser (to get credential) but not to download the corresponding file (.pdf or .tiff or .jpeg). I am using Python 2.7, selenium 3.0.1 and Chrome version 54.0.2840.99 (and chromedriver.exe) on Windows 7 Laptop.
# Chrome web browser.
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
#options.add_argument('--disable-download-notification') #doesn't seems to work
#options.add_experimental_option("prefs", {"download.default_directory","C:\Users\xxx\downloads\Test"}) # doesn't work
#options.add_experimental_option("prefs", {"download.prompt_for_download": False}) # doesn't seems to work
#options.add_experimental_option("prefs", {'profile.default_content_settings': {'images': 2}})# this will disable image loading in the browser
options.add_argument("user-agent="+user_agent_profile)
driver_main = webdriver.Chrome(chrome_options=options)
# Opening the web application portail.
driver_main.get("https://my_link")
I found many discussions on this topic but none of the solution works. For example:
add_experimental_option("prefs", {"download.default_directory","C:\Users\xxx\downloads\Test"})
doesn't work for me.
Same for:
add_experimental_option("prefs", {"download.prompt_for_download": False})
(I also try with "false").
While:
add_argument("user-agent="+user_agent_profile)
Seems to work!
I am not sure to understand what is wrong
The issue I got is that, it starts to download the file each time I open a link with name file(1) file(2) .... file(99) then starting at 100 it opens a popup window "Save As". So I would like to either don't download the file at all or be able to move it in a specific folder in the "Recycle Bin".
How do I find which options could be I used with add_argument and add_argument? I tried to look at Chrome://about/ but I couldn't see a direct correspondence.
Thanks a lot.
Cheers.
Fabien.
The path you declared for the default directory is invalid. Either escape the back slashes or provide a literal string.
options = webdriver.ChromeOptions()
options.add_experimental_option("prefs", {
"download.default_directory": r"C:\Users\xxx\downloads\Test",
"download.prompt_for_download": False,
"download.directory_upgrade": True,
"safebrowsing.enabled": True
})
driver = webdriver.Chrome(chrome_options=options)
Here are the available preferences:
https://cs.chromium.org/chromium/src/chrome/common/pref_names.cc
It makes all the difference in the world to use the forward slash "/" when specifying the directory in which you want things to be downloaded.
I'm guessing this is because that directory will be exported to something like the Powershell, where the usual backslash "\" won't properly work.

R Markdown with Shiny Server change host parameter

I am running RStudio on a server and I created a RMarkdown (.Rmd) file. It works fine if I create it as a static HTML but it does not work if I want it to be interactive (by adding runtime:shiny).
The issue is that when I add runtime:shiny and press the Run Document button the application will try to open at 127.0.0.1:xxxx (here xxxx is a random port). In order to make it work I would have to be able to change the host parameter to '0.0.0.0'. This is an option in the runApp function from the shiny package but I don't know how to add this option in RMarkdown.
Can anyone help me with this?
Thank you.
The ::run command from rmarkdown invokes shiny::runApp internally. You can set the option shiny.host before running the document:
options(shiny.host="0.0.0.0")
rmarkdown::run("myfile.Rmd")
You an also pass arbitrary paramters to runApp, so this should work too:
rmarkdown::run("myfile.Rmd", shiny_args=list(host="0.0.0.0"))
Neither of these will work with the Run Document button; that button starts a new R session in which to render the document. To change the shiny.host option in that session, you'll need to add the option to your .Rprofile.
Set the default values you want to initialize in (~/.Rprofile) under user directory
Sys.setenv(TZ = "UTC") # for Timezone
options(shiny.port = 9999)