Skip to content

Parallel requests for query and diffs

Maximilian Oliver Fisch requested to merge parallel_requests into master
  • utils.add_osm_data(df, path=f'{../../datasets/merged/batches_with_way_ids/unfaelle_berlin_{index}_osm.csv', radius=5) can be called like before but instead of sync request it initiates 10 tcp handshakes at a time (keep-alive) and then proceeds to do 10 http request / tcp connection.
  • For diff query:
    1. Add all queries up into a list

      query_list = []
      for element in group_df:
          adiff = '[adiff:\"2018-01-01T00:00:00Z\",\"2021-12-31T00:00:00Z\"]'
          query_list.append({'params':{'data':f'{adiff}[out:xml];way({element});out meta;'}})
    2. Get your respones collected:

          responses = asyncio.run(utils.overpass_query_parallel_text(query_list))
      Notice overpass_query_parallel_text collects all request that have not received a response with a status code of 200 and retries them in another tcp connection. It only returns if all responses been successful - otherwise a timeout will occur.

Merge request reports