I'm new to Python but required to write a script for API so trying to read the response from API and put it in a file, version is python2.7
Following is the code
import requests
import json
#URL = "someurl"
# sending get request and saving the response as response object
#response = requests.get(url = URL)
#print(response.status_code)
#print(response.content)
items = json.loads('{"batch_id":"5d83a2d317cb4","names":
{"19202":"text1","19203":"text2"}}')
print(items['names'])
for item in items['names']:
print(item)
Current output is
19202
19203
But I would like to pick text1,text2 and write to a file, can anyone help how to get those values
items is a dictionary. items['names'] is also a dictionary. for item in items['names']: will iterate over keys not values. The item will hold key in the dictionary.
To access the value in that key-value pair, you have to use print items['names'][item] instead of print item. Your code should look something like below.
import requests
import json
#URL = "someurl"
# sending get request and saving the response as response object
#response = requests.get(url = URL)
#print(response.status_code)
#print(response.content)
items = json.loads('{"batch_id":"5d83a2d317cb4","names":
{"19202":"text1","19203":"text2"}}')
print(items['names'])
for item in items['names']:
print(items['names'][item])
>>> print(list(items['names'].values()))
['text1', 'text2']
So you can do like this:
for item in items['names'].values():
print(item)
Related
Here is my Django view:
def sendMail(request):
url = 'https://example.com/example'
dictwithlist = request.FILES['dictwithlist']
parsed = json.load(dictwithlist)
// the log file shows that the lists are intact after json.loads
logip(request, json.loads(parsed))
x = requests.post(url, json.loads(clockoutJSON))
return HttpResponse(status=204)
If I just send parsed data my express server receives an empty dict, {}. When I log the json.loads(parsed) I find good data, with the lists intact. When the data gets to the other side though, the dictionaries inside the nested list are all removed, replaced by only strings of their keys.
I tried using headers as described here: Sending list of dicts as value of dict with requests.post going wrong but I just get 500 errors. I don't know if I'm formatting the headers wrong or not. (because the code has line spacing and I'm copying it)
Can anyone help me understand why this is failing? I need that list to get through with its dictionaries intact.
I believe you may need to use dumps rather than loads when sending the request:
x = requests.post(url, json.dumps(parsed))
Alternatively if you're using the python requests library you can send json as a dict as below:
response = requests.post(url=url, json=parsed)
I am testing an endpoint in the following manner:
from rest_framework.test import APIClient
from django.urls import reverse
import json
client = APIClient()
response = client.post(list_url, {'name': 'Zagyg Co'})
I find that the model object is being created with a name of [Zagyg Co] instead of Zagyg Co.
Inspecting the request object reveals the following:
self._request.META['CONTENT_TYPE']
#=> 'multipart/form-data; boundary=BoUnDaRyStRiNg; charset=utf-8'
self._request.body
#=> b'--BoUnDaRyStRiNg\r\nContent-Disposition: form-data; name="name"\r\n\r\nZagyg Co\r\n--BoUnDaRyStRiNg--\r\n'
self._request.POST
#=> <QueryDict: {'name': ['Zagyg Co']}>
Using JSON like so:
response = client.post(
list_url,
json.dumps({'name': 'Zagyg Co'}),
content_type='application/json',
)
sets the name correctly. Why is this so?
request.data is a Django QueryDict. When the data is sent as a multipart form it handles potential multiple values of the same field by putting it in a list.
Using its dict method or using dictionary access syntax returns the last value stored in the relevant key(s):
request.data['name']
#=> 'Zagyg Co'
request.dict()
#=> {'name': 'Zagyg Co'}
Which is great if it's guaranteed that each key has a single value. For list values there's getlist:
request.data.getlist('name')
#=> ['Zagyg Co']
For mixes of keys with single and multiple values it seems like manual parsing is required.
I am stuck with a silly problem.
I have a json data and trying to save it in my model.
Here is the code.
response = response.json() #this gives json data
response = json.loads(response) #loads string to json
json_string = response #ready to get data from list
modelfielda = json_string.get("abc") # this works fine
modelfieldb = json_string.get('["c"]["d"]["e"]') #this does not give data though data is present.
My json data comes like this:
{
"abc":"AP003",
"c":[
{
"d":{
"e":"some data",
"f":"some data"
}
}
]
}
So my question is how to get data inside c.
Try this for e:bnm = json_string.get('c').get('d').get('e') or with list:
bnm = json_string.get('c')[0].get('d').get('e')
By using multiple .gets:
bnm = json_string.get('c')[0].get('d').get('e') # bnm = 'some data'
Or perhaps better (since it will error in case the key does not exists):
bnm = json_string['c'][0]['d']['e'] # bnm = 'some data'
Since you converted it to a Python dictionary, you basically work with a dictionary, and you can obtain the value corresponding to a key by using some_dict[some_key]. Since we here have a cascade of dictionaries, we thus obtain the subdictionary for which we again obtain the corresponding value. The value corresponding to c is a list, and we can obtain the first element by writing [0].
I am new to python as well as scrapy.
I am trying to crawl a seed url https://www.health.com/patients/status/.This seed url contains many urls. But I want to fetch only urls that contain Faci/Details/#somenumber from the seed url .The url will be like below:
https://www.health.com/patients/status/ ->https://www.health.com/Faci/Details/2
-> https://www.health.com/Faci/Details/3
-> https://www.health.com/Faci/Details/4
https://www.health.com/Faci/Details/2 -> https://www.health.com/provi/details/64
-> https://www.health.com/provi/details/65
https://www.health.com/Faci/Details/3 -> https://www.health.com/provi/details/70
-> https://www.health.com/provi/details/71
Inside each https://www.health.com/Faci/Details/2 page there is https://www.health.com/provi/details/64
https://www.health.com/provi/details/65 ... .Finally I want to fetch some datas from
https://www.health.com/provi/details/#somenumber url.How can I achieve the same?
As of now I have tried the below code from scrapy tutorial and able to crawl only url that contains https://www.health.com/Faci/Details/#somenumber .Its not going to https://www.health.com/provi/details/#somenumber .I tried to set depth limit in settings.py file.But it doesn't worked.
import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from news.items import NewsItem
class MySpider(CrawlSpider):
name = 'provdetails.com'
allowed_domains = ['health.com']
start_urls = ['https://www.health.com/patients/status/']
rules = (
Rule(LinkExtractor(allow=('/Faci/Details/\d+', )), follow=True),
Rule(LinkExtractor(allow=('/provi/details/\d+', )),callback='parse_item'),
)
def parse_item(self, response):
self.logger.info('Hi, this is an item page! %s', response.url)
item = NewsItem()
item['id'] = response.xpath("//title/text()").extract()
item['name'] = response.xpath("//title/text()").extract()
item['description'] = response.css('p.introduction::text').extract()
filename='details.txt'
with open(filename, 'wb') as f:
f.write(item)
self.log('Saved file %s' % filename)
return item
Please help me to proceed further?
To be honest, the regex-based and mighty Rule/LinkExtractor gave me often a hard time. For simple project it is maybe an approach to extract all links on page and then look on the href attribute. If the href matches your needs, yield a new Response object with it. For instance:
from scrapy.http import Request
from scrapy.selector import Selector
...
# follow links
for href in sel.xpath('//div[#class="contentLeft"]//div[#class="pageNavigation nobr"]//a').extract():
linktext = Selector(text=href).xpath('//a/text()').extract_first()
if linktext and linktext[0] == "Weiter":
link = Selector(text=href).xpath('//a/#href').extract()[0]
url = response.urljoin(link)
print url
yield Request(url, callback=self.parse)
Some remarks to your code:
response.xpath(...).extract()
This will return a list, maybe you want to have a look on extract_first() which provide the first item (or None).
with open(filename, 'wb') as f:
This will overwrite the file several times. You will only gain the last item saved. Also you open the file in binary mode ('b'). From the filename I guess you want to read it as text? Use 'a' to append? See open() docs
An alternative is to use the -o flag to use scrapys facilities to store the items to JSON or CSV.
return item
It is a good style to yield items instead of return them. At least if you need to create several items from one page you need to yield them.
Another good approach is: Use one parse() function for one type/kind of page.
For instance every page in start_urls fill end up in parse(). From that you extract could extract the links and yield Requests for each /Faci/Details/N page with a callback parse_faci_details(). In parse_faci_details() you extract again the links of interest, create Requests and pass them via callback= to e.g. parse_provi_details().
In this function you create the items you need.
I have link(127.0.0.1:8000/read) which contains json data.how to read a json data from link and pass that values to django form.
Your question is very rough - and without understanding what your requirements are, its hard to place a read example. But this is a simple example of how to get some json:
import urllib2
import json
req = urllib2.Request('http://127.0.0.1:8000/read')
r = urllib2.urlopen(req)
data = r.read()
j = json.loads(data)
node = j['node']['child_node']
for element in node:
print "Element node information: %s" % element['child_node']
#An example of saving information to send to your form
form_text = j['parentnode']['copy']['formtext']