[Tutor] Scraping Wikipedia Table (The retruned file is empty)

[Cynthia Alice Andrews]

At this point I feel like I am wasting my time by not asking for help. I
can't figure out why the file keeps coming back empty. There are no error
message, just an empy file. Very frustrating.

from BeautifulSoup import BeautifulSoup
import urllib2
import csv

wiki = "https://en.wikipedia.org/wiki/List_of_Golden_Globe_winners"
header = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(wiki,headers=header)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)

drama = ""
musical_comedy = ""
drama_actor = ""
musical_comedy_actor = ""
drama_actress = ""
musical_comedy_actress = ""
director = ""

table = soup.find("table", {"class": "wikitable"})

f = open('output.csv','w')

for row in table.findAll("tr"):
cells = row.findAll("td")
if len(cells) == 7:
drama = cells[0].find(text=True)
musical_comedy = cells[1].find(text = True)
drama_actor = cells[2].find(text=True)
musical_comedy_actor = cells[3].find(text=True)
drama_actress = cells[4].find(text=True)
musical_comedy_actress = cells[5].find(text=True)
director = cells[6].find(text=True)

for x in range(len(drama_actor)):
drama_actor_list = drama_actor[x].split("/")
for i in range(len(drama_actor_list)):
if (len(drama_actor_list[i]) > 2) and (len(drama_actor_list[i]) <=5):
data = drama + "," + musical_comedy + "," +
drama_actor_list[i].lstrip('\n').strip() + "," + musical_comedy_actor + ","
+ drama_actress + "," + musical_comedy_actress + "," + director + "," + "\n"
print write_to_file
f.write(write_to_file)
f.close()


*cynthia*andrews
MCDM Candidate, Communication Leadership
University of Washington
<http://www.linkedin.com/in/cynthiaaliceandrews>
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Danny Yoo]

On Thu, Oct 22, 2015 at 4:01 PM, Cynthia Alice Andrews

Post by Cynthia Alice Andrews
At this point I feel like I am wasting my time by not asking for help. I
can't figure out why the file keeps coming back empty. There are no error
message, just an empy file. Very frustrating.

Unfortunately, the formatting of your code broke a bit. Can you try
showing us the code again, and make sure that it's in text mode?

On a first glance, I suspect that the code isn't actually even getting
to the point that it's supposed to write to files. Here's why:
there's a reference to a variable "write_to_file", and I can't tell
where that variable is supposed to be set. That, plus the fact that
you haven't seen any error messages, most likely means at least two
things:

1. You probably need to correct the variable from "write_to_file" to
"data", if I read the intention of the program correctly.

2. You probably need to also correct a bug in the control flow,
because your program isn't reaching the point where it's trying to
write. Otherwise, you would have seen a runtime error when Python
tries to use the name "write_to_file" which hasn't been defined
anywhere.



Looking more at the code...
This looks unusual. drama_actor is just a single string: ranging
across the characters of a string looks very unusual. Are you sure
you want to do this?
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Danny Yoo]

I figured it out! Thanks for taking the time to respond!

No problem; glad you found it. :)
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Mark Lawrence]

Post by Danny Yoo
On Thu, Oct 22, 2015 at 4:01 PM, Cynthia Alice Andrews

Post by Cynthia Alice Andrews
At this point I feel like I am wasting my time by not asking for help. I
can't figure out why the file keeps coming back empty. There are no error
message, just an empy file. Very frustrating.

Looking more at the code...
This looks unusual. drama_actor is just a single string: ranging
across the characters of a string looks very unusual. Are you sure
you want to do this?

A better question IMHO is "where did you learn to write code like that
in the first place", as I've seen so many examples of this that I cannot
understand why people bother writing Python tutorials, as they clearly
don't get read?
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Alan Gauld]

Post by Mark Lawrence

Post by Danny Yoo
Looking more at the code...
This looks unusual...

A better question IMHO is "where did you learn to write code like that
in the first place", as I've seen so many examples of this that I cannot
understand why people bother writing Python tutorials, as they clearly
don't get read?

I think its just a case of bad habits from other languages being
hard to shake off. If your language doesn't have a for-each operator
then its hard to wrap your brain around any other kind of for loop
than one based on indexes.

It's a bit like dictionaries. They are super powerful but beginners
coming from other languages nearly always start out using
arrays(ie lists) and trying to "index" them by searching which
is hugely more complex, but it's what they are used too.

JavaScript programmers tend to think the same about Python
programmers who insist on writing separate functions for
call backs rather than just embedding an anonymous function.
But Python programmers are used to brain dead lambdas with
a single expression so they don't tend to think about
embedding a full function. Familiarity with an idiom makes
it easier to stick with what you know than to try something new.
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[bruce]

my $0.02 for what it might be worth..

You have some people on the list who are straight out begineers who
might be doing cut/copy/paste from 'bad code'. You have people coming
from other languages.. and then you have some who are trying to 'get
through' something, who aren't trying to be the dev!! And yeah, more
time, could easily (in most cases) provide an answer, but sometimes,
you just want to get a soln, and move on to the other 99 probs (no
offense jay z!!)

you guys have been a godsend at times!

thanks - keep up the good fight/work.

Post by Alan Gauld

Post by Mark Lawrence

Post by Danny Yoo
Looking more at the code...
This looks unusual...

A better question IMHO is "where did you learn to write code like that
in the first place", as I've seen so many examples of this that I cannot
understand why people bother writing Python tutorials, as they clearly
don't get read?

I think its just a case of bad habits from other languages being
hard to shake off. If your language doesn't have a for-each operator then
its hard to wrap your brain around any other kind of for loop
than one based on indexes.
It's a bit like dictionaries. They are super powerful but beginners coming
from other languages nearly always start out using
arrays(ie lists) and trying to "index" them by searching which
is hugely more complex, but it's what they are used too.
JavaScript programmers tend to think the same about Python
programmers who insist on writing separate functions for
call backs rather than just embedding an anonymous function.
But Python programmers are used to brain dead lambdas with
a single expression so they don't tend to think about
embedding a full function. Familiarity with an idiom makes
it easier to stick with what you know than to try something new.
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
http://www.flickr.com/photos/alangauldphotos
_______________________________________________
https://mail.python.org/mailman/listinfo/tutor

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor