[Tutor] Help error 504

[Gonzalo V]

how can simulate or emulate an error 504?
i am new in python and its very intuitive! but i am in problems with that
code.

i wrote this code and it cant handle 504 error:
import urllib.request
import urllib.error
from bs4 import BeautifulSoup
import re, csv
from FuncionCsv import LlenarCsv



fhand=open('isbn.txt')
#csvfile=open('ResultadoScrapping.csv', 'w', newline='')
for line in fhand:
req=urllib.request.urlopen('XXXXXXXX'+line)
resp=req.read()
soup=BeautifulSoup(resp,'html.parser')
try:
origen=soup.find(string=re.compile("Origen:
")).find_next().get_text()
nombre=soup.find(name="h1",itemprop="name").get_text()
precioAhora=soup.find(name="p",class_="precioAhora").get_text()
d=soup.find(name="p",class_="stock").get_text()
disp=d.split()
except AttributeError:
disp="no encontrado"
nombre=''
origen=''
precioAhora=''
except urllib.error.HTTPError as e:
if e.getcode()==504:
disp = "sin respuesta del servidor"
print (e.getcode(),disp)
csvfile.close()

print(line,nombre,origen,precioAhora,disp)
line1=line.split()
LlenarCsv('Resultado.csv',line1,nombre,origen,precioAhora,disp)

please help!




Saludos,
Gonzalo
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Danny Yoo]

In your program, you have a try/ except block, but it does not surround the
line:

req=urllib.request.urlopen('XXXXXXXX'+line)

You probably should modify the extent of the exception handling to include
that part. If you are seeing a 504, I expect it to come at this point.
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Laura Creighton]

I use mock for this.

See this blog post which explains it much clearer than I could.

http://engineroom.trackmaven.com/blog/real-life-mocking/

Laura

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Peter Otten]

Post by Gonzalo V
i am new in python and its very intuitive! but i am in problems with that
code.

As Danny already explained an exception can only be caught if the function
is inside a try ... except ...:

try:
req = urllib.request.urlopen(...)
except urllib.error.HTTPError as e:
...

Post by Gonzalo V
how can simulate or emulate an error 504?

(1) You can set up a server under your own control and temporarily use that:

$ cat serve_error.py
import bottle

@bottle.route("/<status:int>")
def set_status(status):
bottle.response.status = status

if __name__ == "__main__":
bottle.run(host="localhost", port=8000)
$ python3 serve_error.py
Bottle v0.12.7 server starting up (using WSGIRefServer())...
Listening on http://localhost:8000/
Hit Ctrl-C to quit.

Post by Gonzalo V

Post by Gonzalo V
import urllib.request
urllib.request.urlopen("http://localhost:8000/504")

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 461, in open
response = meth(req, response)
File "/usr/lib/python3.4/urllib/request.py", line 571, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.4/urllib/request.py", line 499, in error
return self._call_chain(*args)
File "/usr/lib/python3.4/urllib/request.py", line 433, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 579, in
http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 504: Gateway Timeout

The server is flexible enough to produce other errors like the well-known

Post by Gonzalo V

Post by Gonzalo V
try: urllib.request.urlopen("http://localhost:8000/404")

... except Exception as e: print(e)
...
HTTP Error 404: Not Found

Post by Gonzalo V
how can simulate or emulate an error 504?

(2) Doing this from within your own script is called "mocking". While Laura
provided a link that probably explains how to do this right and how unit
tests help avoiding that the same or similar problems will resurface in the
future I'll provide you with the dead simple core of mocking:

You have a function that sometimes shows unwanted behaviour? Temporarily
replace it with another function that always shows this behaviour -- until
the surrounding code has learnt to deal with it.

Post by Gonzalo V
import urllib.request
import urllib.error
from bs4 import BeautifulSoup
import re, csv
from FuncionCsv import LlenarCsv

def raise_httperror_504(url):
raise urllib.error.HTTPError(
url, 504,
"I'm afraid I can't do this", None, None)

urllib.request.urlopen = raise_httperror_504

Post by Gonzalo V
fhand=open('isbn.txt')
#csvfile=open('ResultadoScrapping.csv', 'w', newline='')
req=urllib.request.urlopen('XXXXXXXX'+line)
resp=req.read()
soup=BeautifulSoup(resp,'html.parser')
")).find_next().get_text()
nombre=soup.find(name="h1",itemprop="name").get_text()
precioAhora=soup.find(name="p",class_="precioAhora").get_text()
d=soup.find(name="p",class_="stock").get_text()
disp=d.split()
disp="no encontrado"
nombre=''
origen=''
precioAhora=''
disp = "sin respuesta del servidor"
print (e.getcode(),disp)
csvfile.close()
print(line,nombre,origen,precioAhora,disp)
line1=line.split()
LlenarCsv('Resultado.csv',line1,nombre,origen,precioAhora,disp)

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Timo]

Post by Gonzalo V
how can simulate or emulate an error 504?

I think using a site like http://httpbin.org/ is a bit easier than mock
or your own server.
Just change your request url to:
http://httpbin.org/status/504

Timo

Post by Gonzalo V
i am new in python and its very intuitive! but i am in problems with that
code.
import urllib.request
import urllib.error
from bs4 import BeautifulSoup
import re, csv
from FuncionCsv import LlenarCsv
fhand=open('isbn.txt')
#csvfile=open('ResultadoScrapping.csv', 'w', newline='')
req=urllib.request.urlopen('XXXXXXXX'+line)
resp=req.read()
soup=BeautifulSoup(resp,'html.parser')
")).find_next().get_text()
nombre=soup.find(name="h1",itemprop="name").get_text()
precioAhora=soup.find(name="p",class_="precioAhora").get_text()
d=soup.find(name="p",class_="stock").get_text()
disp=d.split()
disp="no encontrado"
nombre=''
origen=''
precioAhora=''
disp = "sin respuesta del servidor"
print (e.getcode(),disp)
csvfile.close()
print(line,nombre,origen,precioAhora,disp)
line1=line.split()
LlenarCsv('Resultado.csv',line1,nombre,origen,precioAhora,disp)
please help!
Saludos,
Gonzalo
_______________________________________________
https://mail.python.org/mailman/listinfo/tutor

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Peter Otten]

Post by Timo

Post by Gonzalo V
how can simulate or emulate an error 504?

I think using a site like http://httpbin.org/ is a bit easier than mock
or your own server.
http://httpbin.org/status/504

Yes, that's nice!

Plus, it's written in Python, and if you want you can run it on your own
server, too.


_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Laura Creighton]

Thanks Timo! I never heard of this one before!

Laura

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Danny Yoo]

Unit tests that depend on external dependencies can be "flaky": they
might fail for reasons that you don't anticipate. I'd recommend not
depending on an external web site like this: it puts load on someone
else, which they might not appreciate.


Making the test not depend on the network is not bad if we take
advantage of functions. Functions let us swap in different parameters
for values.

Since the code depends on something that knows how to open urls, we
can let that url opener itself be a parameter. The original code
looked something like this:

#######################
for line in fhand:
req=urllib.request.urlopen('XXXXXXXX'+line)
resp=req.read()
....
########################


We can wrap this whole block into a function that takes in a urlopen() function:

####################################
def process(urlopen=urllib.request.urlopen):
for line in fhand:
req=urlopen('XXXXXXXX'+line)
resp=req.read()
....


process()
####################################

where we get the same behavior by calling "process()" at the end. By
default, if we don't give process() an explicit urlopen, it'll use the
urllib.request.urlopen, just like before.


But now it's relatively easy to substitute with something that can
simulate errors:

##########################################
def simulateErrorOnOpen(url):
raise ValueError("oh!")
##########################################

and now we can pass this to our process() to see how it behaves when
url opening raises that error:

###############################
process(urlopen=simulateErrorOnOpen)
###############################
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Peter Otten]

Post by Danny Yoo
Unit tests that depend on external dependencies can be "flaky": they
might fail for reasons that you don't anticipate. I'd recommend not
depending on an external web site like this: it puts load on someone
else, which they might not appreciate.

If you have a well-defined error like an exception raised in a specific
function by all means write a unit test that ensures that your code can
handle it. Whether you use unittest.mock or the technique you describe below
is a matter of taste.

Even if your ultimate goal is a comprehensive set of unit tests a tool like
httbin has its merits as it may help you find out what actually happens in
real life situtations.

Example: Will a server error raise an exception in urlopen() or is it
sometimes raised in the request.read() method?

Also, mocking can give you a false sense of security. coverage.py may report
100% coverage for a function like

def process(urlopen=urllib.request.urlopen):
result = []
for url in [1, 2]:
try:
r = urlopen(url)
result.append(r.read())
except urllib.error.HTTPError:
pass
return result

which will obviously fail once you release it into the wild.

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor