[Tutor] inserting path to open a file from a variable

[richard kappler]

This is a continuation of the read data script I asked for help on
yesterday, which works very well thanks to all the help from the list.

My script opens and reads in new lines from an in service log file,
extracts specific data, writes it to another file for analysis. All of that
works fine, tested out great.

Now I've been tasked to change the script so that the script doesn't need
to be in the same directory as the log file, which makes perfect sense.
Furthermore, the path can't be hard coded into the script, but rather
should read the installer should be able to edit a text file to specify the
paths to the read file (log from which we're extracting data) and the write
file (file to which we're send the extracted data). I thought this would be
a trivial exercise, but I'm stuck.

This is python 2.6.6 running on a Linux machine.

I've created a config file which the user would edit named fileMonitor.conf:

# line two is the absolute path to the log you are parsing data from
# keep 'rdfile:' as is, path starts after it, no spaces
rdfile:Documents/MyScripts/fileMonitor/log.txt
# line 4 is the absolute path to the log you are appending the parsed data
too
# keep 'wrtfile:' as is, path starts after it, no spaces
wrtfile:Documents/MyScripts/fileMonitor/newlog.txt

and I have written up a script that reads the paths and strips off the
unusable bit named readConfig.py:

# read the config file to get file locations for a script
conf = open('fileMonitor.conf', 'r')
read_it = conf.read()

for line in read_it.splitlines():
if line.startswith('rdfile:'):
rd = line
elif line.startswith('wrtfile:'):
wrt = line

rd1 = rd.replace('rdfile:',"",1)
wrt1 = wrt.replace('wrtfile:',"",1)

This worked fine, if I print rd1 and wrt1 I get just the paths that are
entered into the conf file.

Now I need to add that into my fileMonitor.py (parses log for data
extraction) so it goes to the two files.

At the moment, I am, for example, opening the file to be read from with a
simple

file = open('log.txt', 'r')

but I need to replace 'log.txt' with rd1 (path and file name to log.txt).

And I'm stumped. rd1 and wrt1 exist, I can print them, and I get what I
expect (path/filename) for example print rd1 gives
me Documents/MyScripts/fileMonitor/log.txt

But how in the heck do I get that into the open() statement?

What I've tried (none worked):
file = open(rd1, 'r')
file = open('rd1', 'r')

and I ran out of ideas.

Help?

regards, Richard

[Alan Gauld]

Post by richard kappler
# line two is the absolute path to the log you are parsing data from
# keep 'rdfile:' as is, path starts after it, no spaces
rdfile:Documents/MyScripts/fileMonitor/log.txt
# line 4 is the absolute path to the log you are appending the parsed data
too
# keep 'wrtfile:' as is, path starts after it, no spaces
wrtfile:Documents/MyScripts/fileMonitor/newlog.txt

You should maybe look at the config parser module and
use a standard file format. You are likely to be adding
more entries into this file...

Post by richard kappler
and I have written up a script that reads the paths and strips off the
# read the config file to get file locations for a script
conf = open('fileMonitor.conf', 'r')
read_it = conf.read()

You should consider using something like

with open('fileMonitor.conf', 'r') as conf:
for file in conf:

Rather rthan reading/splitting the file in memory.

Post by richard kappler
rd1 = rd.replace('rdfile:',"",1)
wrt1 = wrt.replace('wrtfile:',"",1)
This worked fine, if I print rd1 and wrt1 I get just the paths that are
entered into the conf file.
file = open('log.txt', 'r')
but I need to replace 'log.txt' with rd1 (path and file name to log.txt).
file = open(rd1, 'r')

This should have worked.

What happened when you tried it? "Did not work" is a tad vague!
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[richard kappler]

Sorry Alan, I think I might have accidentally replied to only you. Love
your Python Projects book btw! Working through it now.

I've looked at, and played with a bit, ConfigParser and yes, it seems that
may be a better idea, thank you.

I still have the issue of how to get the path into my code, regardless of
whether I use ConfigParser or

Post by richard kappler
file = open(rd1, 'r')

This should have worked.

What happened when you tried it? "Did not work" is a tad vague!

Traceback (most recent call last):
File "21FileMonitor.py", line 28, in <module>
file = open(rd1, 'r')
IOError: [Errno 2] No such file or directory:
'Documents/MyScripts/fileMonitor/log.txt'

Should I not be using os.path or somesuch? If my path to the file, as
provided via ConfigParser, is rd1, would I do:

file = os.path.open(%s, 'r') % (rd1)
or
file = os.path.open(rd1, 'r')

or do I not need os.path...

Kind of lost in the woods here.

regards, Richard

Post by richard kappler
I've created a config file which the user would edit named

Post by richard kappler
# line two is the absolute path to the log you are parsing data from
# keep 'rdfile:' as is, path starts after it, no spaces
rdfile:Documents/MyScripts/fileMonitor/log.txt
# line 4 is the absolute path to the log you are appending the parsed data
too
# keep 'wrtfile:' as is, path starts after it, no spaces
wrtfile:Documents/MyScripts/fileMonitor/newlog.txt

You should maybe look at the config parser module and
use a standard file format. You are likely to be adding
more entries into this file...
and I have written up a script that reads the paths and strips off the

Post by richard kappler
# read the config file to get file locations for a script
conf = open('fileMonitor.conf', 'r')
read_it = conf.read()

You should consider using something like
Rather rthan reading/splitting the file in memory.
rd1 = rd.replace('rdfile:',"",1)

Post by richard kappler
wrt1 = wrt.replace('wrtfile:',"",1)
This worked fine, if I print rd1 and wrt1 I get just the paths that are
entered into the conf file.
file = open('log.txt', 'r')
but I need to replace 'log.txt' with rd1 (path and file name to log.txt).
file = open(rd1, 'r')

This should have worked.
What happened when you tried it? "Did not work" is a tad vague!
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
http://www.flickr.com/photos/alangauldphotos
_______________________________________________
https://mail.python.org/mailman/listinfo/tutor

[Alan Gauld]

CCing tutor list.
ASlways use ReplyAll (or ReplyList ) to include the list.

Post by richard kappler
file = open(rd1, 'r')
This should have worked.
What happened when you tried it? "Did not work" is a tad vague!
File "21FileMonitor.py", line 28, in <module>
file = open(rd1, 'r')
'Documents/MyScripts/fileMonitor/log.txt'

Notice that path is a relative path not an absolute one.
Its looking for a folder called Documents in the current directory.
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Felix Dietrich]

Post by richard kappler
Now I've been tasked to change the script so that the script doesn't need
to be in the same directory as the log file, which makes perfect sense.
Furthermore, the path can't be hard coded into the script, but rather
should read the installer should be able to edit a text file to specify the
paths to the read file (log from which we're extracting data) and the write
file (file to which we're send the extracted data). I thought this would be
a trivial exercise, but I'm stuck.

An alternative way to configurate execution parameters is to pass the
filenames as arguments to your program which you can access via
/sys.argv/ (modules to consider: /getopt/, /optparser/):

import sys
print sys.argv[0] # the first element is the script's name
print sys.argv[1] # second element first argument (e.g. rd)
print sys.argv[2] # third element second argument (e.g. wd)

If you have these lines in a file "argv_test.py" try:

python argv_test.py read_filename write_filename


Another good alternative for filters is to simply read from stdin and
output the results to stdout; then use the shell to redirect those
from/to the respective files:

python filter.py <rdfile >wdfile
# < redirects stdin
# > redirects stdout
# or with a pipe and cat
cat rdfile | python filter.py > wdfile

Within python stdin can be read via sys.stdin:

import sys
for l in sys.stdin:
print l


The /ConfigParser/-Module provides a way to read and write configuration
files.

Post by richard kappler
# read the config file to get file locations for a script
conf = open('fileMonitor.conf', 'r')
read_it = conf.read()
rd = line
wrt = line

Instead of reading all the content of a file into memory one can simply
iterate it and retrieve the contents line my line (I believe this way is
also considered more "pythonic"):

conf = open('fileMonitor.conf', 'r')
for line in conf:
...

One more "pythonic" thing to do is to wrap the interaction with a file
in a /with/-block: that way file closing is ensured after one is done
with the file:

with open('fileMonitor.conf', 'r') as conf:
for line in conf:
...

Post by richard kappler
At the moment, I am, for example, opening the file to be read from with a
simple
file = open('log.txt', 'r')
but I need to replace 'log.txt' with rd1 (path and file name to log.txt).
And I'm stumped. rd1 and wrt1 exist, I can print them, and I get what I
expect (path/filename) for example print rd1 gives
me Documents/MyScripts/fileMonitor/log.txt
But how in the heck do I get that into the open() statement?
file = open(rd1, 'r')
file = open('rd1', 'r')

What do you mean by "none worked"? Did python respond with an error?
How did you figure that the calls to /open/ failed?

Also: The first line opens a file having the path of the string the
variable /rd1/ currently holds (presumably
"Documents/MyScripts/fileMonitor/log.txt"). The second calls /open/
with the string "rd1" causing /open/ to try and open a file with the
name rd1.

'r' will fail when the file does not exist.

--
Felix Dietrich
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[richard kappler]

I found the problem, but the answer confuses me.

If I run my script to open a file in Documents/MyScripts/fileMonitor which
is where I'm doing my building and testing, with the variable rd1 (however
created, my way and ConfigParser way both work) from within
Documents/MyScripts/fileMonitor, the script fails with a

Traceback (most recent call last):
File "21FileMonitor.py", line 28, in <module>
file = open(rd1, 'r')
IOError: [Errno 2] No such file or directory:
'Documents/MyScripts/fileMonitor/log.txt'

but if I run the exact same script from the Home directory, it works fine,
does exactly what I expected it to do (ie. opens the files).

I thought it might be because because I used Doc... instead of ~/Doc... for
my path, but I got the same traceback.

curiouser and curiouser was, Richard

On Thu, May 28, 2015 at 3:01 PM, Felix Dietrich <

Post by Felix Dietrich

Post by richard kappler
Now I've been tasked to change the script so that the script doesn't need
to be in the same directory as the log file, which makes perfect sense.
Furthermore, the path can't be hard coded into the script, but rather
should read the installer should be able to edit a text file to specify

the

Post by richard kappler
paths to the read file (log from which we're extracting data) and the

write

Post by richard kappler
file (file to which we're send the extracted data). I thought this would

be

Post by richard kappler
a trivial exercise, but I'm stuck.

An alternative way to configurate execution parameters is to pass the
filenames as arguments to your program which you can access via
import sys
print sys.argv[0] # the first element is the script's name
print sys.argv[1] # second element first argument (e.g. rd)
print sys.argv[2] # third element second argument (e.g. wd)
python argv_test.py read_filename write_filename
Another good alternative for filters is to simply read from stdin and
output the results to stdout; then use the shell to redirect those
python filter.py <rdfile >wdfile
# < redirects stdin
# > redirects stdout
# or with a pipe and cat
cat rdfile | python filter.py > wdfile
import sys
print l
The /ConfigParser/-Module provides a way to read and write configuration
files.

Post by richard kappler
# read the config file to get file locations for a script
conf = open('fileMonitor.conf', 'r')
read_it = conf.read()
rd = line
wrt = line

Instead of reading all the content of a file into memory one can simply
iterate it and retrieve the contents line my line (I believe this way is
conf = open('fileMonitor.conf', 'r')
...
One more "pythonic" thing to do is to wrap the interaction with a file
in a /with/-block: that way file closing is ensured after one is done
...

Post by richard kappler
At the moment, I am, for example, opening the file to be read from with a
simple
file = open('log.txt', 'r')
but I need to replace 'log.txt' with rd1 (path and file name to log.txt).
And I'm stumped. rd1 and wrt1 exist, I can print them, and I get what I
expect (path/filename) for example print rd1 gives
me Documents/MyScripts/fileMonitor/log.txt
But how in the heck do I get that into the open() statement?
file = open(rd1, 'r')
file = open('rd1', 'r')

What do you mean by "none worked"? Did python respond with an error?
How did you figure that the calls to /open/ failed?
Also: The first line opens a file having the path of the string the
variable /rd1/ currently holds (presumably
"Documents/MyScripts/fileMonitor/log.txt"). The second calls /open/
with the string "rd1" causing /open/ to try and open a file with the
name rd1.
'r' will fail when the file does not exist.
--
Felix Dietrich
_______________________________________________
https://mail.python.org/mailman/listinfo/tutor

[Felix Dietrich]

Post by richard kappler
If I run my script to open a file in Documents/MyScripts/fileMonitor which
is where I'm doing my building and testing, with the variable rd1 (however
created, my way and ConfigParser way both work) from within
Documents/MyScripts/fileMonitor, the script fails with a
File "21FileMonitor.py", line 28, in <module>
file = open(rd1, 'r')
'Documents/MyScripts/fileMonitor/log.txt'
but if I run the exact same script from the Home directory, it works fine,
does exactly what I expected it to do (ie. opens the files).
I thought it might be because because I used Doc... instead of ~/Doc... for
my path, but I got the same traceback.

"Documents/MyScripts/fileMonitor" is a relative path (does not start
with a '/'): it will be looked-up starting from the current working
directory.

Assuming the following directory tree:

/
└── home
└── richard
└── Documents
└── MyScripts
└── fileMonitor
├── fileMonitor.py
└── log.txt

and that you are inside the fileMonitor directory ('$' indicates
commands typed at a shell prompt):

$ pwd
/home/richard/Documents/MyScripts/fileMonitor

Now when you try to open a file with the path

"Documents/MyScripts/fileMonitor/log.txt"

from within the fileMonitor directory you are trying to open a file at
(note that I use the '\' as line continuation marker)

"/home/richard/Documents/MyScripts/fileMonitor/Documents/MyScripts/\
cwd_example/log.txt"

To open "/home/richard/Documents/MyScripts/fileMonitor/log.txt" with the
current directory set to "/home/richard/Documents/MyScripts/fileMonitor"
you can refer to it as "log.txt" (also "../fileMonitor/log.txt" and all
sorts of less sensible addressing).

If your path on the other hand starts with a '/' it is an absolute path
starting with the directory tree root: the path
"/home/richard/Documents/MyScripts/cwd_example/log.txt" will always
refer to "/home/richard/Documents/MyScripts/cwd_example/log.txt" no
matter the current working directory.


Another example:

/
└── home
└── felix
└── Documents
└── MyScripts
└── cwd_example
└── cwd

Here is the content of cwd:

#!/usr/bin/python
import os
print(os.getcwd())

Make it executable:

$ pwd
/home/felix/Documents/MyScripts/cwd_example/
$ chmod +x cwd

Note that it is not unusual for a script to drop its file extension when
it is made executable; the language used to implement is not important
anymore and the executable becomes not distinguishable by extension from
other executables whether written in an interpreted language or binary.

The shebang ("#!") line acts as magic to identify the interpreter to
use.

Some examples:

$ pwd
/home/felix/Documents/MyScripts
$ cwd_example/cwd
/home/felix/Documents/MyScripts
$ /home/felix/Documents/MyScripts/cwd_example/cwd
/home/felix/Documents/MyScripts

$ pwd
/home/felix/Documents/MyScripts/cwd_example/log-dir
$ ../cwd # .. = parent directory
/home/felix/Documents/MyScripts/cwd_example/log-dir
$ /home/felix/Documents/MyScripts/cwd_example/cwd
/home/felix/Documents/MyScripts/cwd_example/log-dir

$ pwd
/home/felix/Documents/MyScripts/cwd_example/
$ ./cwd # . = current directory
/home/felix/Documents/MyScripts/cwd_example/
$ /home/felix/Documents/MyScripts/cwd_example/cwd
/home/felix/Documents/MyScripts/cwd_example/

Again the absolute path works independently of the current working
directory. ".." refers to the previous directory; "." to the current
one. And the working directory is not related to the directory the
executable resides in – though they might happen to be the same.

The following only works when either '.' or
"/home/felix/Documents/MyScripts/cwd_example/" are part of the PATH
environment variable:

$ pwd
/home/felix/Documents/MyScripts/cwd_example/
$ cwd

Regarding the '~': I do not think that (most?) python functions
automatically expand the tilde to the user's directory; instead look at

Post by richard kappler

os.path.expanduser("~/Documents")

'/home/felix/Documents'

Have a look at the modules /os/ and /os.path/ (also /pathlib/ for >=3.4)
functions /os.chdir/, /os.getcwd/, /os.expanduser/ specifically.

/sys.path/ is also related to the current working directory and relative
paths and it will come in handy when you import your own modules not
residing in any of the default directories while running the script with
a different working directory than its location.

--
Felix Dietrich
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
http

[Mark Lawrence]

Post by Felix Dietrich

Post by richard kappler
Now I've been tasked to change the script so that the script doesn't need
to be in the same directory as the log file, which makes perfect sense.
Furthermore, the path can't be hard coded into the script, but rather
should read the installer should be able to edit a text file to specify the
paths to the read file (log from which we're extracting data) and the write
file (file to which we're send the extracted data). I thought this would be
a trivial exercise, but I'm stuck.

An alternative way to configurate execution parameters is to pass the
filenames as arguments to your program which you can access via

optparse is deprecated, from
https://docs.python.org/3/library/optparse.html "Deprecated since
version 3.2: The optparse module is deprecated and will not be developed
further; development will continue with the argparse module". argparse
is here https://docs.python.org/3/library/argparse.html

There are also the third party modules docopt
https://github.com/docopt/docopt which is excellent and clize
https://github.com/epsy/clize which I've never tried.
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Felix Dietrich]

Post by Mark Lawrence
optparse is deprecated, from
https://docs.python.org/3/library/optparse.html "Deprecated since
version 3.2: The optparse module is deprecated and will not be
developed further; development will continue with the argparse
module". argparse is here
https://docs.python.org/3/library/argparse.html

True, but the /optparse/ module does not appear to be part of Python
2.6. ("new in version 3.2")

Post by Mark Lawrence

Post by richard kappler
This is python 2.6.6 running on a Linux machine.

--
Felix Dietrich
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Mark Lawrence]

Post by Felix Dietrich

Post by Mark Lawrence
optparse is deprecated, from
https://docs.python.org/3/library/optparse.html "Deprecated since
version 3.2: The optparse module is deprecated and will not be
developed further; development will continue with the argparse
module". argparse is here
https://docs.python.org/3/library/argparse.html

True, but the /optparse/ module does not appear to be part of Python
2.6. ("new in version 3.2")

If you mean argparse you're correct, but it's in 2.7. My point is that
there's not much use writing code now with a deprecated module,
especially if you want to move onward and upward to python 3.4 or even
3.5 in the future and get all the new batteries :)

Post by Felix Dietrich

Post by Mark Lawrence

Post by richard kappler
This is python 2.6.6 running on a Linux machine.

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Felix Dietrich]

Post by Mark Lawrence

Post by Felix Dietrich
True, but the /optparse/ module does not appear to be part of Python
2.6. ("new in version 3.2")

If you mean argparse you're correct, but it's in 2.7. My point is
that there's not much use writing code now with a deprecated module,
especially if you want to move onward and upward to python 3.4 or even
3.5 in the future and get all the new batteries :)

Yes, i meant /argparse/ and I should have checked the documentation more
thoroughly; I only noticed that it was not part of 2.6 and saw the note
at the top of the /Python 3/ documentation.

I agree in general that it is better to avoid using deprecated modules.
Still, if one is stuck with 2.6 – one does not always get the choice of
new fancier batteries. ;)

--
Felix Dietrich
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Brandon McCaig]

Post by richard kappler
# line two is the absolute path to the log you are parsing data from
# keep 'rdfile:' as is, path starts after it, no spaces
rdfile:Documents/MyScripts/fileMonitor/log.txt
# line 4 is the absolute path to the log you are appending the parsed data
too
# keep 'wrtfile:' as is, path starts after it, no spaces
wrtfile:Documents/MyScripts/fileMonitor/newlog.txt

Your problem traces all the way back to here. The comments describe
the configuration file as expecting absolute paths, but you have only
entered relative paths (absolute paths need to be "rooted"; in *nix
that typically means a prefix of /). A good option then is to validate
your configuration file when you read it in.

Post by richard kappler
# read the config file to get file locations for a script
conf = open('fileMonitor.conf', 'r')
read_it = conf.read()
rd = line
wrt = line
rd1 = rd.replace('rdfile:',"",1)
wrt1 = wrt.replace('wrtfile:',"",1)

import os.path
import sys

for x in [(rd1, 'Source file', 'rdfile'),
(wrt1, 'Destination file', 'wrtfile')]:
if not os.path.isabs(x[0]):
print >> sys.stderr, \
'Configuration error:', \
'%s path (%s) must be absolute!' % \
x[1:]
sys.exit(1)

Regards,
--
Brandon McCaig <***@gmail.com> <***@castopulence.org>
Castopulence Software <https://www.castopulence.org/>
Blog <http://www.bambams.ca/>
perl -E '$_=q{V zrna gur orfg jvgu jung V fnl. }.
q{Vg qbrfa'\''g nyjnlf fbhaq gung jnl.};
tr/A-Ma-mN-Zn-z/N-Zn-zA-Ma-m/;say'
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor