Schedule Tasks

How to Display Date

The date command displays the current date and time. It has several options:

mm/dd/yy is displayed with

$ date "+%D"
# OUTPUT
09/24/12

$ date "+%T"
# OUTPUT
12:03:47

Here are some more:

Specifier	Explanation
`%d`	Displays the day of the month (01 to 31)
`%h`	Displays the abbreviated month name (Jan to Dec)
`%m`	Displays the month of year (01 to 12)
`%Y`	Displays the four-digit year
`%T`	Displays the time in 24 hour format as HH:MM:SS
`%H`	Displays the hour

Schedule Job

crontab

cron table: A Linux system file that creates a table-like structure where fields are separated by white space. Users can populate the table by assigning values to each field (asterisk).

As you can see, the crontab syntax has 5 asterisks. Here’s what each of those asterisk represent:

	1st	2nd	3rd	4th	5th
	*	*	*	*	*
ID	Minute	Hour	Day-Date	Month	Day Name
Values	0-59	0 -23	1-31	1-12	0-6

NOTE: Day Names 0-6 begin with Sunday.
The time field uses 24 hours format
In order to schedule a task, you replace the appropriate asterisk with your desired value.

# ___  To run the job at 00:00 every Sunday
0 0 * * 0

# ___  To run every minute
* * * * *

# ___  To run every hour
0 * * * *
        
# ___  To run every 4pm on all Sundays in October
0 16 * 10 0

# ___  To run every 1st of the month
0 0 1 * *
        
# ___  To run on June 10th at 8:30am
30 08 10 06 * /path/full-backup

-e edit the crontab file
-l list existing cronjobs
-r -i remove default crontab, after asking for confirmation
-u specify a different user’s crontab

Schedule more than once

The comma-separated value in a field specifies that the command needs to be executed at all the mentioned times.
The following script takes an incremental backup twice a day every day. This example executes the specified incremental backup shell script (incremental-backup) at 11:00 and 16:00 every day.

# ___  To run at 11:00 and 16:00 every day
00 11, 16 * * * /path/incremental-backup

# ___  To run every day during the hours of 9am - 6pm
00 09-18 * * * /path/check-db-status

# ___  To run every weekday during the hours of 9am - 6pm
00 09-18 * * 1-5 /path/check-db-status

# ___  To run every 10 minutes
*/10 * * * *   /path/check-db-status

# ___ Append date/time to a file every Sunday at 6:15pm
15 18 * * 0 date >> sundays.txt

# ___ Run a script on the first minute of the first day of each month
1  0 1 * * ./My_Shell_Script.sh

# ___ Back up your home directory every Monday at 3:00 am
0 3 * * 1  tar -cvf my_backup_path\my_archive.tar.gz $HOME\

@

There are special cases in which instead of the above 5 fields you can use @ followed by a keyword — such as reboot, midnight, yearly, or hourly.

Keyword	Equivalent
@yearly	0 0 1 1 *
@daily	0 0 * * *
@hourly	0 * * * *
@reboot	Run at startup

Here is a table detailing how to schedule a DAG (Apache Airflow Pipeline)

Alternatively, you can also use one of these cron “preset”:

preset	meaning	cron
`None`	Don’t schedule, use for exclusively “externally triggered” DAGs
`@once`	Schedule once and only once
`@hourly`	Run once an hour at the beginning of the hour	`0 * * * *`
`@daily`	Run once a day at midnight	`0 0 * * *`
`@weekly`	Run once a week at midnight on Sunday morning	`0 0 * * 0`
`@monthly`	Run once a month at midnight of the first day of the month	`0 0 1 * *`
`@yearly`	Run once a year at midnight of January 1	`0 0 1 1 *`

Note: Use schedule_interval=None and not schedule_interval='None' when you don’t want to schedule your DAG.

Cron

Cron is a system daemon used to execute desired tasks in the background at designated times.

A crontab file is a simple text file containing a list of commands meant to be run at specified times. It is edited using the crontab command.
Each line in a crontab file has five time-and-date fields, followed by a command, followed by a newline character (\n).
The fields are separated by spaces.
The five time-and-date fields cannot contain spaces and their allowed values are as follows:

Field	Allowed values
minute	0-59
hour	0-23, 0 = midnight
day	1-31
month	1-12
weekday	0-6, 0 = Sunday

List Cron Jobs

The -l option of the crontab command prints the current crontab.

$ crontab -l
no crontab for xyzdje

Create a job that runs echo when the minute is 0 and the hour is 21. It effectively means the job runs at 9.00 p.m every day.

Add job in Crontab

crontab -e adds a cron job

This will create a new crontab file for you
An editor is opened and shown for you to add your cron job
The comment lines in the file explain how to do this
Add this line at the end of the file
The line specifies that the echo command should run when the minute is 0 and the hour is 21. It effectively means the job runs at 9.00 p.m every day.
The output of the command should be sent to a file /tmp/echo.txt
Press Ctrl + x to save the changes
Press y to confirm.

$ crontab -e
# add this line at the end of the file
0 21 * * * echo "Welcome to cron" >> /tmp/echo.txt

After saving you get this image
Press Enter to come out of the editor
List cron jobs to make sure it is saved

$ crontab -l

# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command
0 21 * * * echo "Welcome to cron" >> /tmp/echo.txt

Schedule a Shell Script -1

Create Script

Let us create a simple shell script that prints the current time and the current disk usage statistics.

Step 1: On the menu on the lab screen, use File->New File to create a new file:
Step 2: Give the file name as diskusage.sh and click ‘OK’
Step 3: Save the following commands into the shell script
Step 4: Save the file using the File->Save menu option
Step 5: Verify that the script is working:

#! /bin/bash
# print the current date time
date
# print the disk free statistics
df -h

# set permission
$ chmod u+x diskusage.sh
# check permissions
$ ls -l diskusage.sh
-rwxr--r-- 1 theia users 86 Sep 25 16:58 diskusage.sh
# execute script
$ ./diskusage.sh
Wed Sep 25 17:02:09 EDT 2012
Filesystem      Size  Used Avail Use% Mounted on
overlay          98G   33G   61G  35% /
tmpfs            64M     0   64M   0% /dev
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/vda2        98G   33G   61G  35% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs            28G   16K   28G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            16G     0   16G   0% /proc/acpi
tmpfs            16G     0   16G   0% /proc/scsi
tmpfs            16G     0   16G   0% /sys/firmware

Schedule job

Let us schedule this script to be run everyday at midnight 12:00 (when the hour is 0 on the 24 hour clock).

We want the output of this script to be appended to /home/project/diskusage.log
First command opens the crontab file
Add this line to the end of the file after the line from above
0 0 * * * /home/project/diskusage.sh >>/home/project/diskusage.log
Press Ctrl + x to save the changes
Press y to confirm
Press Enter to come out of the editor
Check if the job is added to the crontab by running the following command
crontab -l
You see the job was added at the end

$ crontab -l

# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command
0 21 * * * echo "Welcome to cron" >> /tmp/echo.txt
0 0 * * * /home/project/diskusage.sh >>/home/project/diskusage.log

Remove Crontab

The -r option causes the current crontab to be removed.

Caution: This removes all your cron jobs. Be extra cautious when you use this command on a production server.

$ crontab -r

# verify crontab was removed
$ crontab -l

no crontab for dingoxy

Schedule a Script - 2

Create a cron job that runs the task date >> /tmp/everymin.txt every minute.

Edit Crontab

$ crontab -e

# Add the following line at the end of the file
* * * * * date >> /tmp/everymin.txt

# save and quit the editor

Run Commands in Parallel

./ETL_chunk_one_on_these_nodes.sh  & ./ETL_chunk_two_on_those_nodes.sh

Run Commands Sequentially

start=$(date); ./MyBigScript.sh ; end=$(date)

Linux Automation Techniques

Refer to this Linux document for more.