79239725

Date: 2024-11-30 14:20:19
Score: 1
Natty:
Report link

Not all websites allow automated access

Hey @MITHU, the reason you are getting ConnectionError is because some websites simply don't allow automated access to prevent from bots. You can check that by using webiste_url/robots.txt Usually they would have something like below:

User-agent: *
Disallow: /

You can try out this working example:

import requests

url = 'https://github.com/'

headers = {
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'accept-language': 'en-US,en;q=0.9',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
}

with requests.Session() as session:
    session.headers.update(headers)

    response = session.get(url)

    print("Status Code:", response.status_code)
    print("Page Snippet:", response.text[:500]) 
Reasons:
  • Long answer (-0.5):
  • Has code block (-0.5):
  • User mentioned (1): @MITHU
  • Low reputation (1):
Posted by: Pratik