Dfyjkt

Suppose I have a list of objects and for each of these objects I want to create a dictionary and put all the dictionaries I generated in a list.

What I am doing is the following:

def f(x):
 some function f that returns a dictionary given x

list_of_dict = 

xlist = [x1, x2, ..., xN]

for x in xlist:
 list_of_dict.append(f(x))

I am wondering whether there is a more efficient (faster) way to create a list of dictionaries than the one I am proposing.

Thank you.

Since you are dealing with HTTP requests (which is not obvious from your question), the rest of the answer is irrelevant: communications will dominate the computations by a sheer margin. I'll leave the answer here, anyway.

The original approach seems to be the slowest:

In [20]: %%timeit 
 ...: list_of_dict = 
 ...: for x in xlist: 
 ...: list_of_dict.append(f(x)) 
 ...: 
13.5 µs ± 39.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Mapping is the best way to go:

In [21]: %timeit list(map(f,xlist)) 
8.45 µs ± 17 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

List comprehension is somewhere in the middle:

In [22]: %timeit [f(x) for x in xlist] 
10.2 µs ± 22.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Since you mentioned in the comments that you want faster execution overall, maybe async the requests?

Check out this example adapted from this blog post: http://skipperkongen.dk/2016/09/09/easy-parallel-http-requests-with-python-and-asyncio/

import asyncio
import requests

async def main():
 xlist = [...]
 list_of_dict = 
 loop = asyncio.get_event_loop()
 futures = [
 loop.run_in_executor(
 None, 
 requests.get, 
 i
 )
 for i in xlist
 ]
 for response in await asyncio.gather(*futures):
 respons_dict = your_parser(response) # Your parsing to dict from before
 list_of_dict.append(response_dict)
 return list_of_dict

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

I think this can help you with implement parallelism. Just note that you should be sure you aren't flooding somebody with requests that they wouldn't appreciate.