Job Distribution

Often there are a number of Python tasks that you need to distribute to multiple servers / instances. Traditionally, you have to declare a central server that distributes jobs and collects results. Scheduling the jobs is a non-trivial task, which often leads to inappropriately used server resources. This setting is not very robust, as everything revolves around the central server, which becomes particularly problematic if it fails.

Pyremto makes the task of distributing Python jobs much easier. You only have to formulate the jobs once via JSON files and upload them to the Pyremto server. You can then distribute your worker scripts to your servers / instances. These can now request tasks from the Pyremto server and send the results back.

You can see how many jobs have already been completed in the app at any time. This way of managing jobs is robust, as processing continues seamlessly even if individual servers / instances fail. The jobs are distributed on-demand, so you don’t have to worry about the runtime of the jobs beforehand. This ensures that all resources can be used to the maximum at all times!

How does that work?

Job scheduling is based on job serialisation as JSON objects. For example, if you want to compute the value of a specific function for different arguments, you could define your jobs as

jobs = [{x: 0.25}, {x: 1.65}, ...]

These jobs are then sent to the pyremto server, from which your worker instances can poll new tasks each time they are ready. This can be done with few lines of code:

remote = RemoteControl(show_qr=True)
job = remote.get_next_job()
result = perform_job(job)
remote.set_job_done(job_id=job["id"], result=result)

You can see a full example how to use the job scheduling here. Moreover, there is an example how you can use the job scheduling for a machine learning hyperparameter search.

Maximize resource utilization by proper job scheduling

By using pyremto job scheduling, it is possible to easily distribute Python tasks to different servers / instances. With on-demand scheduling, you can maximize resource usage so that you can get the most out of your available resources.