WOLFRAM

Wolfram Language & System Documentation Center
Wolfram Compute Services Batch Computation

Wolfram Compute Services Batch Computation

Setting Up a Batch JobUsing Paclets and Resources in a Batch Job
NotificationsParallel Computation
Batch Job OptionsIncremental Output

Wolfram Compute Services provides a way to submit batch jobs using cloud compute resources.

RemoteBatchSubmit[expr]submit a batch job
job["JobStatus"]enquire about a batch job's status
job["EvaluationResult"]obtain a job's result

Submitting batch jobs and obtaining their status and results.

A batch job is submitted and then runs unattended. You can check its progress repeatedly and be notified when it completes. Once complete, you can obtain its result. Batch jobs are ideal for long-running jobs and for jobs that require large compute resources.

Submit a job to search for Mersenne primes:
At any time, ask for the job status:
After a while, the job will be running:
After the job has succeeded, you can download its result:

Setting Up a Batch Job

Typically, you may want to develop your code using a small example, then submit a larger example as a batch job.

Write a function with a search range parameter:
Test it locally with a small example:

Once the code seems to be working, you can submit a larger instance. Definitions, such as for findMersennePrime, that you make in your session will be distributed automatically to the batch job, which is a completely separate Wolfram session. Only in this way will the batch job know what to do with findMersennePrime.

Submit a search job with a given upper bound:
Once the job has succeeded, you can download its result:
Obtain metadata about the job, such as the time it took it to complete:

Notifications

By default, a batch job will send two email messages to the address, which is your $WolframID. One message is sent when the job begins running, another one when it completes.

The message sent when the job completes successfully will contain a piece of code you can evaluate to retrieve the job's result. This code works in any Wolfram session; you do not need to keep the session in which you submitted the job active.

Evaluate the code from the email message to retrieve a job's result:

Given the job's UUID, you can retrieve the job object in a new session and continue to work with it.

Given the job's UUID, you can retrieve the job object in a new session:
All of the job's properties are available again:

Notifications can be customized with an option setting in RemoteBatchSubmit.

Nonedo not send any notifications
JobCompletedsend a single message when the job completes
{"Email","SMS"}send notifications both by email and SMS
{"JobCreditsUsed",n}send notifications every time n more credits are spent

Common settings for the RemoteJobNotifications option.

Do not send any notifications for a small test job:
Verify the exact notification setting used in a job:
The job submitted earlier used the default set of notifications:

Batch Job Options

Job Limits

You can limit a job's maximal runtime, or maximal cost, expressed in Service Credits. A job that does not succeed before a limit is reached will be aborted, returning a "Terminated" failure status.

Limit a job's run time to 60 seconds:
This job would take much longer, and it will be aborted after running for approximately 60 seconds.
An aborted job will not return a result:
You can find more details about why a job was terminated:
It does make a log file available:

Choosing Compute Resources

The default available remote machine class, "Basic1x8", is the most cost effective and provides one CPU and a limited amount of memory. It is ideal for development and smaller tasks that do not benefit form parallelism.

At the highest end, "Compute192x384" provides 192 CPUs and 384GB of memory.

Parallel Computation

You can choose to run a job on a remote machine with many processors and run a parallel computation. Parallel kernels will be launched automatically.

Submit a parallel search job on a suitable machine class with 64 processors:
After the job has succeeded, you can download its result:
Any print output ends up in the job's log file:
You can obtain additional information about the job's performance, such as its run time:

Incremental Output

A remote batch job that terminates abnormally will not return a result. For long-running search jobs that return only a few results, it may make sense to capture the results as they are found, while the job is still running.

You can use a cloud object to accumulate the results as they are found.

Initialize a cloud object that represents an empty list:
Append data to the cloud object:
The cloud object collects all appended data in a list:

In the search job, each result is appended to the cloud object as it is found, and regular progress is printed to the log file. The job is set up without an upper limit on the search range, running until it is terminated either when it reaches a limit or when it is aborted.

The credit limit safeguards against spending too many credits, and regular notifications about credits spent provide feedback:

Once the job is running, results start to show up in the cloud object:
The job's log file shows how far the search progressed:
Once you have found the desired results, you can abort the job without losing the results:
The results are still available in the cloud object: