The only obvious difference between the requests that succeeded and the ones that failed was that the failed requests had a Content-Encoding of 'br' (Brotli). It's possible this is somehow connected, since there are known issues related to support for Brotli in the requests library:
# Seems to work consistently
r.server_ip = r.raw._original_response.fp.raw._sock.getpeername()[0]
r.server_port = r.raw._original_response.fp.raw._sock.getpeername()[1]
Download more RAM. 🐏 ⨉ 0Posted by August R. Garcia 1 year ago
Edit History
• [2019-09-25 15:54 PDT] August R. Garcia (1 year ago) • [2019-09-25 15:54 PDT] August R. Garcia (1 year ago) • [2019-09-25 15:54 PDT] August R. Garcia (1 year ago) 🕓 Posted at 25 September, 2019 15:54 PM PDT
Hire freelancers and/or advertise your goods and/or services. Hire people directly. We're not a middleman or your dad. Manage your own business transactions.
Answers in Web Scraping, Data Analysis | By August R. Garcia
Published | Last Update
618 views, 0 RAMs, and 0 comments
Tags: Python, Urllib, Brotli, urllib3, requests
The solution here seems to generally work:
However, there seems to be some edge case where this occassionally doesn't work. See these debugging statements:
The only obvious difference between the requests that succeeded and the ones that failed was that the failed requests had a Content-Encoding of 'br' (Brotli). It's possible this is somehow connected, since there are known issues related to support for Brotli in the requests library:
Anyway, using this version instead of the code at the start of this post seems to work consistently:
Edit History
• [2019-09-25 15:54 PDT] August R. Garcia (1 year ago)• [2019-09-25 15:54 PDT] August R. Garcia (1 year ago)
• [2019-09-25 15:54 PDT] August R. Garcia (1 year ago)
🕓 Posted at 25 September, 2019 15:54 PM PDT
Sir, I can do you a nice SEO.
Post a New Comment
To leave a comment, login to your account or create an account.
Do you like having a good time?
Register an Account
You can also login to an existing account or reset your password. All use of this site is subject to the terms of service and privacy policy.
Read Quality Articles
Read some quality articles. If you can manage to not get banned for like five minutes, you can even post your own articles.
View Articles →
Argue with People on the Internet
Use your account to explain why people are wrong on the Internet forum.
View Forum →
Vandalize the Wiki
Or don't. I'm not your dad.
View Wiki →
Ask and/or Answer Questions
If someone asks a terrible question, post a LMGTFY link.
View Answers →
Make Some Money
Hire freelancers and/or advertise your goods and/or services. Hire people directly. We're not a middleman or your dad. Manage your own business transactions.
Answers— Read More
Find more related content below!