Dfyjkt

I have a function which returns a 2 Dimensional numpy array and this function will be inside a loop. At each iteration, I want to append this numpy array into a file.

filename = "xyz"
for i in range(10):
 np_array = function_to_get_numpy()
 now append this `np_array` into filename

I can continue to append the numpy array in the script and dump once, but I want to avoid that.

Also I would prefer to store this in non-binary format.

In [64]: with open('xyz','w') as f:
 ...: for n in range(1,4):
 ...: arr = np.arange(n*n).reshape(n,n)
 ...: np.savetxt(f, arr, fmt='%5d', delimiter=',')
 ...: 
In [65]: cat xyz
 0
 0, 1
 2, 3
 0, 1, 2
 3, 4, 5
 6, 7, 8

If the number of columns varies, as it does here, it will be hard(er) to read. csv readers like genfromtxt won't like it.

If the number of columns is consistent, it can be loaded as one big array. Separating the writes and reloading them is possible, but more involved.

I'm going to take the opportunity to plug nppretty, a pretty printer for numpy that I've been working on. It provides a class ArrayStream that I think will do exactly what you need.

Install nppretty with:

pip install nppretty

You can use ArrayStream much like a file object. For example, this code:

from nppretty import ArrayStream
import numpy as np

arrstr = ArrayStream('arraystream.txt', name='arr2D')
for i in range(10):
 arr = np.arange(10*i, 10*(i + 1))
 arrstr.write(arr.reshape(2,5))
arrstr.close()

will produce a text file called arraystream.txt with the following contents:

arr2D = [
[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19],
[20, 21, 22, 23, 24],
[25, 26, 27, 28, 29],
[30, 31, 32, 33, 34],
[35, 36, 37, 38, 39],
[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59],
[60, 61, 62, 63, 64],
[65, 66, 67, 68, 69],
[70, 71, 72, 73, 74],
[75, 76, 77, 78, 79],
[80, 81, 82, 83, 84],
[85, 86, 87, 88, 89],
[90, 91, 92, 93, 94],
[95, 96, 97, 98, 99],
]

Notes on ArrayStream

ArrayStream accepts all of the same arguments as the standard Python open method. The one additional keyword arg is name, which sets the name of the array in the file (name defaults to "array" if left blank). Just like the file object returned by a call to open, an ArrayStream instance can used in a with statement. For example, the following code will produce the same output as above:

with ArrayStream('arraystream.txt', name='arr2D') as f:
 for i in range(10):
 arr = np.arange(10*i, 10*(i + 1))
 f.write(arr.reshape(2,5))