[Tutor] Request for help with os.walk() combining os.path.ismount() in Linux

[Srihari Vijayaraghavan]

Hello Folks,

Please consider the following 2 scripts:
1. os_path_ismount.py:
import os
import sys
out = sys.stdout.write
out("%s\n" % os.path.ismount(sys.argv[1]))

2. os_walk.py:
import sys
import os
out = sys.stdout.write
for root, dirs, files in os.walk("/"):
out("The dirs before removing mount points: %s\n" % dirs)
for d in dirs:
dname = os.path.join(root, d)
if os.path.ismount(dname):
dirs.remove(d)
out("The dirs after removing mount points: %s\n" % dirs)
sys.exit()

Am at a loss to understand this quirky behaviour on Fedora 22 (x86-64;
both stock-standard python 2.7.10 & 3.4.2):
[***@laptop ~]$ python2 ./os_path_ismount.py /proc
True

[***@laptop ~]$ python3 ./os_path_ismount.py /proc
True

[***@laptop ~]$ python2 ./os_walk.py
The dirs before removing mount points: ['run', 'dev', 'srv', 'fedora',
'root', 'bin', 'lib', 'opt', 'lost+found', 'etc', 'sbin', 'var',
'sys', 'media', 'backup', 'home', 'usr', 'tmp', 'proc', 'mnt', 'boot',
'lib64']
The dirs after removing mount points: ['dev', 'srv', 'fedora', 'root',
'bin', 'lib', 'opt', 'lost+found', 'etc', 'sbin', 'var', 'media',
'backup', 'usr', 'proc', 'mnt', 'lib64']

[***@laptop ~]$ python3 ./os_walk.py
The dirs before removing mount points: ['run', 'dev', 'srv', 'fedora',
'root', 'bin', 'lib', 'opt', 'lost+found', 'etc', 'sbin', 'var',
'sys', 'media', 'backup', 'home', 'usr', 'tmp', 'proc', 'mnt', 'boot',
'lib64']
The dirs after removing mount points: ['dev', 'srv', 'fedora', 'root',
'bin', 'lib', 'opt', 'lost+found', 'etc', 'sbin', 'var', 'media',
'backup', 'usr', 'proc', 'mnt', 'lib64']

Undoubtedly proc (to be precise /proc) (same for /dev as well) is
indeed a mount point, yet it's somehow not being evicted. So am I
doing something silly? Could somebody please explain what am I doing
wrong here?

However, /run & /sys are being evicted, which is good, of course.

(Interestingly, I get different results on different platforms:
CentOS6.7 gives some results, CentOS7.1 something else, i.e., am
totally baffled!)

Thank you.
Srihari Vijayaraghavan

PS: Of course, I could add an exclusion list to take out the "known"
mount points, but that won't be elegant I reckon. Certainly, it won't
be for my real usage scenario.
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Alan Gauld]

Post by Srihari Vijayaraghavan
out = sys.stdout.write
out("The dirs before removing mount points: %s\n" % dirs)
dname = os.path.join(root, d)
dirs.remove(d)

It's never a good idea to remove items from the thing
you are iterating over. Create a copy of dirs (dirs[:])
to iterate on then remove the items from the original
dirs.

<aside>
It always amazes me how often the same issues come up in
rapid succession. We must have had 4 or 5 of these types
of errors in the last couple of months, and other times
we go for 6 months or more without it being raised! :-)
</aside>
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Laura Creighton]

Post by Alan Gauld
<aside>
It always amazes me how often the same issues come up in
rapid succession. We must have had 4 or 5 of these types
of errors in the last couple of months, and other times
we go for 6 months or more without it being raised! :-)
</aside>

There are only 17 people in the world. We simulate the rest with
mirrors. :)

Laura

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Peter Otten]

Post by Laura Creighton

Post by Alan Gauld
<aside>
It always amazes me how often the same issues come up in
rapid succession. We must have had 4 or 5 of these types
of errors in the last couple of months, and other times
we go for 6 months or more without it being raised! :-)
</aside>

There are only 17 people in the world. We simulate the rest with
mirrors. :)

You have that backwards: there are only 17 mirrors in the world; we simulate
the rest with people.

;)

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Srihari Vijayaraghavan]

Post by Alan Gauld

Post by Srihari Vijayaraghavan
out = sys.stdout.write
out("The dirs before removing mount points: %s\n" % dirs)
dname = os.path.join(root, d)
dirs.remove(d)

It's never a good idea to remove items from the thing
you are iterating over. Create a copy of dirs (dirs[:])
to iterate on then remove the items from the original
dirs.

In general I agree, but this is what the os.walk() document states:
"... When topdown is True, the caller can modify the dirnames list
in-place (perhaps using del or slice assignment)..."

(Yes, the topdown argument of os.walk() is True by default, until
manually modified.)

Therefore I see no problem with in-place modification of dirnames.
I've made a copy of dirs & iterated over it, which made no difference
in my case.

My issue is that some of the virtual or in memory file systems' mount
points (like /proc, /dev) are failing to be recognised properly under
os.path.ismount(). Perhaps this quirkiness is irresolvable??

Thank you.

Srihari Vijayaraghavan
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Srihari Vijayaraghavan]

On 20 August 2015 at 18:51, Srihari Vijayaraghavan

Post by Srihari Vijayaraghavan

Post by Alan Gauld

Post by Srihari Vijayaraghavan
out = sys.stdout.write
out("The dirs before removing mount points: %s\n" % dirs)
dname = os.path.join(root, d)
dirs.remove(d)

It's never a good idea to remove items from the thing
you are iterating over. Create a copy of dirs (dirs[:])
to iterate on then remove the items from the original
dirs.

"... When topdown is True, the caller can modify the dirnames list
in-place (perhaps using del or slice assignment)..."
(Yes, the topdown argument of os.walk() is True by default, until
manually modified.)
Therefore I see no problem with in-place modification of dirnames.
I've made a copy of dirs & iterated over it, which made no difference
in my case.

Sorry to reply to my own email.

I stand corrected. Indeed, while iterating over dirs (in the above
example) & doing in-place modification was the source of the problem,
giving unpredictable outcome. While iterating over its copy & updating
the original dirs, gives expected results.

Perhaps my interpretation of the document wasn't correct. Anyway, it's
working now.

Thank you folks, especially Alan.

[Alan Gauld]

Post by Srihari Vijayaraghavan
"... When topdown is True, the caller can modify the dirnames list
in-place (perhaps using del or slice assignment)..."

That's true so far as the os.walk call goes.
That is, altering dirs does not have any nagative
impact on the subsequent iterations of os.walk.
So even if you delete items from dirs os.walk will
continue to iterate over those deleted directories
in subsequent calls to os.walk()

But altering dirs within the current iteration does affect
dirs within that iteration, just like any other collection.
So your for loop inside the os.walk loop is affected by the
deletions even if the outer os.walk loop is not.
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Srihari Vijayaraghavan]

Post by Alan Gauld

Post by Srihari Vijayaraghavan
"... When topdown is True, the caller can modify the dirnames list
in-place (perhaps using del or slice assignment)..."

That's true so far as the os.walk call goes.
That is, altering dirs does not have any nagative
impact on the subsequent iterations of os.walk.
So even if you delete items from dirs os.walk will
continue to iterate over those deleted directories
in subsequent calls to os.walk()

Thanks Alan. That's now practically learnt :-).

Post by Alan Gauld
But altering dirs within the current iteration does affect
dirs within that iteration, just like any other collection.
So your for loop inside the os.walk loop is affected by the
deletions even if the outer os.walk loop is not.

Yes, it was a bad idea to update dirs list while iterating over it. I
was confused/mislead by the documentation, perhaps by misinterpreting
it. Never mind, now it's quite clear that it's a no-no.

Thanks your quick & perfect analysis.

Srihari Vijayaraghavan
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor