Two great questions. Xcalar Design has Jupyter Notebook built in for you to apply machine learning models or non-native operations.
We call deployment logic "batch dataflows". You can apply a batch dataflow via CI/CD by first scheduling an execution through the interface, then creating a BASH shell script from the cron entry.
You can schedule a batch dataflow as follows:
1. Click the Dataflows icon .
2. From the Dataflows Panel, select your batch dataflow for scheduling.
3. In the Batch Dataflows window, above and to the right of the Dataflow Graph, click the Add schedule to dataflow icon .
4. Select the *Advanced Mode tab.
5. In the Schedule field, enter a cron format string to specify execution times.
6. Click SIMULATE.
7. Confirm the Last Run and expected Next Run date/time
8. Click SAVE.
Here's how you create the BASH shell script:
- Display a list of all cron entries using the following command:
- Copy the line from your crontab containing the name of your Batch Dataflow. It will look similar to the CRON Entry Example below.
- Create a new file named trigger_batchDataflowName.sh, and paste the cron line into it.
- Delete the cron syntax expression for the execution time, for example:
22 2 31 10 * 2018.
- Add the following line to the beginning of the file:
- This line tells a UNIX-like operating system to run the file as a BASH shell script.
The shell script should look like the Shell Script Example, below.
- Make the script file executable using the chmod command:
chmod a+x triggerbatchDataflowName.sh_
- Remove the cron entry used in the script file.
- Enter in the following command:
- Use your preferred text or visual editor to remove the dataflow entry.
- Save and exit the editor.
- Verify the cron entry is gone by listing the cron entries as in step 2, with the following command:
Note: You remove the cron entry to prevent the scheduled run from executing. Do not pause the scheduled event in Xcalar, as this will prevent the shell script-triggered run from executing.
Now, you are ready to configure Jenkins or another third-party scheduling tool to trigger the trigger_batchDataflowName.sh file and therefore run your Batch Dataflow. Running a Batch Dataflow adds a log entry to your Run History in the Schedule Details panel in the Batch Dataflow Window.
Cron Entry Example
22 2 31 10 * 2018 (export XLRDIR=/opt/xcalar; export PYTHONPATH=; /opt/xcalar/bin/python /opt/xcalar/scripts/scheduleRetinas/executeFromCron/executeScheduledRetina.py batchDataflowName_XcalarScheduledRetina) 2>&1 | /usr/bin/logger -t ScheduledRetina_batchDataflowName
Shell Script Example
(export XLRDIR=/opt/xcalar; export PYTHONPATH=; /opt/xcalar/bin/python /opt/xcalar/scripts/scheduleRetinas/executeFromCron/executeScheduledRetina.py batchDataflowName_XcalarScheduledRetina) 2>&1 | /usr/bin/logger -t ScheduledRetina_batchDataflowName
Did that answer your question?