[Tutor] How to read all integers from a binary file?

[David Aldrich]

Hi

I have a binary file of 32-bit unsigned integers. I want to read all those integers into a list.

I have:

ulData = []
with open(ulBinFileName, "rb") as inf:
ulData.append(struct.unpack('i', inf.read(4)))

This obviously reads only one integer from the file. How would I modify this code to read all integers from the file please?

Best regards

David

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Peter Otten]

Post by David Aldrich
Hi
I have a binary file of 32-bit unsigned integers. I want to read all those
integers into a list.
ulData = []
ulData.append(struct.unpack('i', inf.read(4)))
This obviously reads only one integer from the file. How would I modify
this code to read all integers from the file please?

While the obvious approach would be a loop

data = [] # no misleading prefixes, please
while True:
four_bytes = inf.read(4)
if len(four_bytes) != 4:
if four_bytes:
raise ValueError("number of bytes not a multiple of 4")
break
data.append(struct.unpack("i", four_bytes))


there is also the array module:

data = array.array("i", inf.read())

You can convert the array to a list with

data = data.tolist()

if necessary.

_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[David Aldrich]

Post by Peter Otten
data = array.array("i", inf.read())

Thanks for all the answers to my question. I used the array method in the end.

Best regards

David
_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Laura Creighton]

Post by David Aldrich
Hi
I have a binary file of 32-bit unsigned integers. I want to read all those integers into a list.
ulData = []
ulData.append(struct.unpack('i', inf.read(4)))
This obviously reads only one integer from the file. How would I modify this code to read all integers from the file please?
Best regards
David

You did the hard part, finding the struct module:
and I assume that you got struct_unpack defined somewhere
You need to wrap that up in:

with open(ulBinFileName, "rb") as inf:
while True:
data = inf.read(4)
if not data:
break
ulData.append(struct.unpack('i', data))

Laura


_______________________________________________
Tutor maillist - ***@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Martin A. Brown]

Hello there David,

Post by David Aldrich
I have a binary file of 32-bit unsigned integers. I want to read
all those integers into a list.
ulData = []
ulData.append(struct.unpack('i', inf.read(4)))
This obviously reads only one integer from the file. How would I
modify this code to read all integers from the file please?

I see you already have responses from Laura Creighton and Peter
Otten. I particularly like the idea of using array.array() to read
the data.

You can also use a number in the format string to indicate how many
times you want to apply the format string.

For example, assume your data file was exactly 20 bytes long and has
been read into memory as a variable called binary_data. Now you
want to make a list containing 5 of these unsigned integers. Well,
you can do this:

l = struct.unpack('5i', binary_data)

# -- that would give you this tuple (fabricated data)
#
# l = (-1682686882, 2011801612, -366700247, 463482871, 1463075426)

Below, you can see what I did to generalize this technique in the
function read_data().

If, however, you do not have complex data, then array.array() is
probably the easiest path for you.

Good luck and have fun!

-Martin

from __future__ import print_function

import sys
import struct


def write_data(f, size, datatype):
data = gen_data(size)
fmt = str(len(data)) + datatype
s = struct.pack(fmt, *data)
f.write(s)
f.close()


def gen_data(size):
import numpy
data = list(numpy.random.randint(-(2**31)+1, 2**31, size=size))
return data


def read_data(f, datatype):
s = f.read()
f.close()
element_size = struct.calcsize(datatype)
q, r = divmod(len(s), element_size)
assert r == 0
fmt = str(int(len(s) / element_size)) + 'i'
data = struct.unpack(fmt, s)
return data


if __name__ == '__main__':
if len(sys.argv) > 2:
size = int(sys.argv[2])
write_data(open(sys.argv[1], 'wb'), size, 'i')
data = read_data(open(sys.argv[1], 'rb'), 'i')
print(len(data))



P.S. You'll see that I didn't have a mess of unsigned integers
hanging around in a file, so you can see how I generated and
stored them in write_data and gen_data).