Shell scripts in Linux — practice with a grader that compiles, tests and marks C submissions (and the assignment that “compiled” without compiling)

In the second week of my IC2 group’s grader, something odd happened. Pablo told me: “hey, your script says my assignment compiles, but I get an error”. And he was right: his .c didn’t compile. What had happened? The day before, an old version had compiled, the pablo executable was still in the folder and my script, instead of checking whether gcc had gone well, was checking whether the executable existed. It was grading the past.

The lesson stuck with me: in a script, what counts is the command’s exit status, not what you think it left in the folder. And, while you’re at it, delete the old stuff before generating the new.

This practice puts everything from the shell scripts theory to work with three medium-level programs that, together, make up a complete assignment grader: compile.sh (compiles every submission and keeps the errors apart), run_tests.sh (runs each one with a fixed input and compares its output) and grade.sh (the final grader, with options, functions and a mark). The data looks like the real thing: five submissions of the “sum” assignment (read two integers and print their sum), and among them there’s one that doesn’t compile, another that compiles with a warning and another that compiles but gives the wrong result.

Before you start

It helps to be clear on:

• How a script is launched: #!/bin/bash, chmod +x and ./script.sh.
• That if checks a command’s exit status (0 = fine) and that [ ] is just another command, with spaces.
• The parameters $1, $# and "$@", and why variables go in quotes.
• The redirections from Redirection and pipes: >, 2>, <, /dev/null and |.

If any of them sounds shaky, go over the theory before carrying on.

Program 1: compile every submission

compile.sh takes a folder, compiles each .c with gcc -Wall and prints one line per student: OK, OK (with warnings) or ERROR. The gcc messages aren’t lost: they go to one file per student in errors/. What makes it tricky: we have to check that the folder was passed correctly, tell warnings apart from errors and make the script return a useful code for whoever uses it.

The idea

The order of the steps is the same as in any serious script:

1. Check the parameters at the start. If there isn’t exactly one, or it isn’t a folder, message to stderr and exit 2. Better to stop at the start than to fail halfway through.
2. Go through the submissions with for source in "$folder"/*.c.
3. For each one: get the student’s name with basename, delete the old executable and compile, sending stderr to errors/<student>.txt.
4. Decide using gcc‘s code (not by checking whether the executable exists): if it’s 0, it compiles; if on top of that the errors file isn’t empty, there were warnings.
5. Count, print the summary and end with a code: 0 if everything compiled, 1 if not.

The code

#!/bin/bash
# compile.sh: compiles every submission in a folder and says which ones compile
# Usage: ./compile.sh folder

if [ $# -ne 1 ]; then                  # exactly one parameter: the folder
    echo "Usage: $0 folder" >&2         # error messages go to stderr
    exit 2                             # 2 = you called me wrongly
fi
folder=$1
if [ ! -d "$folder" ]; then            # -d: exists and is a folder
    echo "Not a folder: $folder" >&2
    exit 2
fi

mkdir -p errors                        # -p: does not fail if it already exists
ok=0
failed=0
for source in "$folder"/*.c; do        # quotes on the variable, the * outside
    student=$(basename "$source" .c)   # submissions/ana.c -> ana
    binary="$folder/$student"
    rm -f "$binary"                    # without this, an old executable would pass for a good one
    if gcc -Wall "$source" -o "$binary" 2> "errors/$student.txt"; then
        ok=$((ok + 1))
        if [ -s "errors/$student.txt" ]; then   # -s: exists and is NOT empty
            echo "OK     $student (with warnings)"
        else
            echo "OK     $student"
        fi
    else
        failed=$((failed + 1))
        echo "ERROR  $student"
    fi
done
echo "Compiled: $ok. Failed: $failed"
[ "$failed" -eq 0 ]                    # last command = script's exit code: 0 if all compiled

The heart of the script is the loop. Let’s take a closer look:

for source in "$folder"/*.c; do        # quotes on the variable, the * outside
    student=$(basename "$source" .c)   # submissions/ana.c -> ana
    binary="$folder/$student"
    rm -f "$binary"                    # without this, an old executable would pass for a good one
    if gcc -Wall "$source" -o "$binary" 2> "errors/$student.txt"; then
        ok=$((ok + 1))
        if [ -s "errors/$student.txt" ]; then   # -s: exists and is NOT empty
            echo "OK     $student (with warnings)"
        else
            echo "OK     $student"
        fi
    else
        failed=$((failed + 1))
        echo "ERROR  $student"
    fi
done

• "$folder"/*.c: the variable goes in quotes (in case the folder has spaces) and the * outside, because bash wouldn’t expand it inside quotes.
• basename "$source" .c removes the folder and, with the second parameter, the extension too: submissions/ana.c → ana.
• rm -f "$binary" is the fix from my story. The -f stops rm from complaining if the file doesn’t exist (the first time it never does).
• if gcc … 2> "errors/$student.txt"; then: compiles and, at the same time, saves the messages. The if checks gcc‘s code.
• [ -s "errors/$student.txt" ]: -s is true if the file exists and isn’t empty. If gcc compiled but wrote something, those are warnings.

This is the trace of the loop with our five submissions:

Round	student	gcc’s $?	errors empty?	Prints	ok	failed
1	ana	0	yes	OK ana	1	0
2	dani	0	yes	OK dani	2	0
3	laura	0	yes	OK laura	3	0
4	pablo	1	—	ERROR pablo	3	1
5	sergio	0	no	OK sergio (with warnings)	4	1

And the last line, [ "$failed" -eq 0 ], prints nothing: it’s the script’s last command, so its code is the script’s code. With failed = 1, it returns 1.

Output

$ chmod +x compile.sh
$ ./compile.sh
Usage: ./compile.sh folder
$ echo $?
2
$ ./compile.sh submissionss
Not a folder: submissionss
$ echo $?
2
$ ./compile.sh submissions
OK     ana
OK     dani
OK     laura
ERROR  pablo
OK     sergio (with warnings)
Compiled: 4. Failed: 1
$ echo $?
1
$ ls submissions
ana
ana.c
dani
dani.c
laura
laura.c
pablo.c
sergio
sergio.c
$ ls errors
ana.txt
dani.txt
laura.txt
pablo.txt
sergio.txt
$ cat errors/pablo.txt
submissions/pablo.c: In function 'main':
submissions/pablo.c:6:26: error: expected ';' before 'return'
    6 |     printf("%d\n", a + b)
      |                          ^
      |                          ;
    7 |     return 0;
      |     ~~~~~~                
$ head -n 2 errors/sergio.txt
submissions/sergio.c: In function 'main':
submissions/sergio.c:5:9: warning: unused variable 'x' [-Wunused-variable]

Let’s check it by hand. No parameters: usage message and code 2. With submissionss (misspelt): not a folder, code 2 again. With submissions: 4 compile (ana, dani, laura and sergio) and 1 fails (pablo), 4 + 1 = 5 submissions, and the script returns 1 because failed isn’t 0.

ls submissions confirms there are four new executables and no pablo: the rm -f would have deleted it even if it existed from before. In errors/ there’s one file per student (ana’s, dani’s and laura’s are empty). Pablo’s says exactly what’s wrong: a ; is missing at the end of line 6. Sergio’s is a warning: the variable x isn’t used. It compiles, but -Wall flags it, and our script marks it as OK (with warnings).

Questions you might ask

Why do the usage messages use >&2? Because they’re errors, not results. If someone runs ./compile.sh submissions > result.txt, the file should only contain the OK and ERROR lines; usage warnings have to show up on screen. It’s the stdout/stderr split you saw in Redirection.

Why exit 2 and not exit 1? It’s a very common convention: 2 usually means “you called me wrongly” (ls, grep, bash -n… do it) and 1, “I did the work, but something failed”. That way, whoever uses the script can tell a usage error apart from a submission that doesn’t compile.

Why not check whether the executable exists? Because of the story at the start: an old executable can still be there. gcc‘s exit status tells you what happened in this run, which is the only thing we care about. And the rm -f is the second safety net.

What happens if the folder has no .c files? Bash leaves the pattern as it is: the loop runs once with source literally being submissions/*.c, and gcc fails. You fix it by putting [ -f "$source" ] || continue at the start of the loop (or with shopt -s nullglob, which makes a pattern with no matches turn into nothing).

Why does the summary say “Failed: 1” instead of stopping at the first error? Because a grader has to grade everyone. An error in pablo’s file is no reason not to compile sergio’s. What we do is remember it in failed and reflect it in the exit status at the end.

Try it yourself

• Add a -q (quiet) option that prints nothing and only returns the exit status, to use it in an if.
• Make the script print, at the end, the first line of every errors file that isn’t empty (hint: head -n 1 inside a for over errors/*.txt).

Program 2: test the outputs

run_tests.sh goes one step further: it runs each submission with the input from input.txt (3 4) and compares what it prints with expected.txt (7). For each student it says PASS, WRONG OUTPUT (and what it got) or NO EXECUTABLE. What makes it tricky: each student’s check has three possible results, not two, and we want it inside a function.

The idea

The run_test function does the work and returns a number; the main loop turns that number into text with a case:

• 2 if there’s no executable (pablo didn’t compile).
• If there is one, it runs it with < input.txt and saves what it prints in outputs/<student>.txt.
• Then it compares with cmp -s, which returns 0 if both files are identical and 1 if not. Since it’s the function’s last command, its code is the function’s code.

Separating “work out” from “print” makes the function reusable: program 3 could use it without changing anything.

The code

#!/bin/bash
# run_tests.sh: runs each submission with input.txt and compares with expected.txt
# Usage: ./run_tests.sh [folder]   (after ./compile.sh folder)

folder=${1:-submissions}             # if no folder is given, use submissions

run_test() {                         # run_test STUDENT -> 0 ok, 1 wrong output, 2 no executable
    local student=$1
    local binary="$folder/$student"
    if [ ! -x "$binary" ]; then      # -x: exists and can be executed
        return 2
    fi
    "$binary" < input.txt > "outputs/$student.txt"
    cmp -s "outputs/$student.txt" expected.txt   # -s: no messages; only its $? matters
}

mkdir -p outputs
for source in "$folder"/*.c; do
    student=$(basename "$source" .c)
    run_test "$student"
    case $? in                       # case checks the code run_test returned
        0) echo "PASS            $student" ;;
        1) echo "WRONG OUTPUT    $student: got $(cat "outputs/$student.txt"), expected $(cat expected.txt)" ;;
        2) echo "NO EXECUTABLE   $student" ;;
    esac
done

The newest bits are in the function:

run_test() {                         # run_test STUDENT -> 0 ok, 1 wrong output, 2 no executable
    local student=$1
    local binary="$folder/$student"
    if [ ! -x "$binary" ]; then      # -x: exists and can be executed
        return 2
    fi
    "$binary" < input.txt > "outputs/$student.txt"
    cmp -s "outputs/$student.txt" expected.txt   # -s: no messages; only its $? matters
}

• folder=${1:-submissions} (line 5 of the script): ${1:-value} means “$1, and if it’s empty, value“. That makes the parameter optional.
• [ ! -x "$binary" ]: -x checks that it exists and can be executed. If not, return 2 and the function ends there.
• "$binary" < input.txt > "outputs/$student.txt": the input comes from the file (the program’s scanf reads 3 4) and the output goes to another file. The usual redirections, but on a program of ours.
• cmp -s: compares byte by byte; the -s (silent) makes it print nothing. We only care about its code.

This is the function’s trace for each student:

student	-x?	Writes to outputs/	cmp -s	Returns	The case prints
ana	yes	7	same	0	PASS
dani	yes	12	different	1	WRONG OUTPUT
laura	yes	7	same	0	PASS
pablo	no	—	—	2	NO EXECUTABLE
sergio	yes	7	same	0	PASS

Output

$ chmod +x compile.sh run_tests.sh
$ ./compile.sh submissions > /dev/null
$ ./run_tests.sh submissions
PASS            ana
WRONG OUTPUT    dani: got 12, expected 7
PASS            laura
NO EXECUTABLE   pablo
PASS            sergio
$ cat outputs/dani.txt
12
$ ls outputs
ana.txt
dani.txt
laura.txt
sergio.txt

The first command compiles with > /dev/null: we don’t want to see its OK lines again, only for it to leave the executables ready. Then the maths: 3 + 4 = 7, and that’s what ana, laura and sergio print. Dani prints 3 × 4 = 12, because it multiplies. Pablo has no executable, so it isn’t even tried. In outputs/ there are four files, not five: pablo’s is never created, because the function does return 2 before running anything.

Questions you might ask

Why cmp -s and not compare with [ "$(cat a)" = "$(cat b)" ]? Both work here. cmp compares the exact files (including the final newlines), whereas $( ) strips trailing newlines. For a strict grader, cmp; for one that forgives an extra newline, the string comparison. In program 3 we use the second.

Why does the case $? go right after run_test "$student"? Because $? is the code of the last command. If there were an echo in between, the case would see the echo‘s 0 and everyone would come out PASS.

Why local? Because student also exists in the main loop. Without local, the function would use the same variable; here it wouldn’t break anything (they hold the same value), but it’s a bug waiting to happen the day you change something. With local, the function touches nothing outside.

What happens if a student submits a program that waits forever? The grader hangs with it. A real grader uses timeout 5 "$binary" < input.txt, which kills it after 5 seconds and returns 124. That would be a fourth case for the case.

Try it yourself

• Add the timeout case: if the program takes longer than 2 seconds, the function returns 3 and the case prints TOO SLOW.
• Make the script test with several inputs: input1.txt/expected1.txt, input2.txt/expected2.txt…, and say how many each student passes.

Program 3: the complete grader

grade.sh puts everything together: it compiles, tests and gives a mark (10 if the output is correct, 5 if it compiles but prints something else, 0 if it doesn’t compile). It also accepts an option: -c to compile only and -p to compile and test (what it does if you say nothing). What makes it tricky: options come before the folder, so the folder may be in $1 or in $2, and the exit status has to say whether all the submissions are right.

The idea

• A usage function with the error message and exit 2, so it isn’t repeated three times.
• A very short compile function whose exit status is gcc‘s.
• A case "$1" for the option. If there’s an option, shift removes it and the folder becomes $1. That way, the rest of the script doesn’t have to ask whether there was an option or not.
• The loop uses continue to jump to the next student as soon as it knows their mark.
• The last command compares correct with total to give the script’s code.

The code

#!/bin/bash
# grade.sh: the full grader for the "sum" assignment
# Usage: ./grade.sh [-c | -p] folder
#   -c  compile only    -p  compile and test (what it does if you say nothing)

usage() {                            # a function so the message is not repeated three times
    echo "Usage: $0 [-c | -p] folder" >&2
    exit 2                           # exit inside a function ends the WHOLE script
}

compile() {                          # compile SOURCE BINARY: returns gcc's code
    gcc -Wall "$1" -o "$2" 2> /dev/null
}

mode=test
case "$1" in
    -c) mode=compile; shift ;;       # shift: drops $1 and the rest moves ($2 becomes $1)
    -p) shift ;;
    -*) usage ;;                     # any other option starting with -
esac
[ $# -eq 1 ] || usage                # || : if the check fails, usage
[ -d "$1" ] || usage
folder=$1

correct=0
total=0
for source in "$folder"/*.c; do
    student=$(basename "$source" .c)
    binary="$folder/$student"
    total=$((total + 1))
    rm -f "$binary"
    if ! compile "$source" "$binary"; then
        echo "$student: 0 (does not compile)"
        continue                     # on to the next student
    fi
    if [ "$mode" = compile ]; then   # = compares STRINGS
        echo "$student: compiles"
        correct=$((correct + 1))
        continue
    fi
    output=$("$binary" < input.txt)
    if [ "$output" = "$(cat expected.txt)" ]; then
        echo "$student: 10"
        correct=$((correct + 1))
    else
        echo "$student: 5 (compiles, but got $output)"
    fi
done
echo "Correct: $correct of $total"
[ "$correct" -eq "$total" ]          # exit code: 0 only if all are right

The hardest part is the options:

mode=test
case "$1" in
    -c) mode=compile; shift ;;       # shift: drops $1 and the rest moves ($2 becomes $1)
    -p) shift ;;
    -*) usage ;;                     # any other option starting with -
esac
[ $# -eq 1 ] || usage                # || : if the check fails, usage
[ -d "$1" ] || usage
folder=$1

• mode=test is the default value: if there’s no option, nothing changes it.
• -c) changes the mode and does shift. -p) only does shift, because the mode was already test.
• -*) matches anything else starting with -: it’s an option we don’t know, so usage. It comes after -c and -p because in a case the first matching pattern wins.
• If $1 doesn’t start with - (for example, submissions), no branch matches and nothing happens.
• [ $# -eq 1 ] || usage: “either there’s exactly one parameter, or usage”. It reads like a one-line if.

Notice that usage ends with exit 2, not return: if the call is wrong, there’s no point carrying on, and exit inside a function ends the whole script.

In the loop, output=$("$binary" < input.txt) saves what the student’s program prints, and [ "$output" = "$(cat expected.txt)" ] compares strings with =. The trace in test mode:

Round	student	Compiles?	output	Mark	correct	total
1	ana	yes	7	10	1	1
2	dani	yes	12	5	1	2
3	laura	yes	7	10	2	3
4	pablo	no	—	0	2	4
5	sergio	yes	7	10	3	5

Output

$ chmod +x grade.sh
$ ./grade.sh -x submissions
Usage: ./grade.sh [-c | -p] folder
$ echo $?
2
$ ./grade.sh -c submissions
ana: compiles
dani: compiles
laura: compiles
pablo: 0 (does not compile)
sergio: compiles
Correct: 4 of 5
$ ./grade.sh submissions | tee report.txt
ana: 10
dani: 5 (compiles, but got 12)
laura: 10
pablo: 0 (does not compile)
sergio: 10
Correct: 3 of 5
$ echo $?
0
$ ./grade.sh submissions > /dev/null
$ echo $?
1
$ ./grade.sh submissions > /dev/null && echo "All correct" || echo "Needs review"
Needs review
$ wc -l report.txt
6 report.txt

Let’s check it. With -x: unknown option, usage and code 2. With -c: compile only, so the 4 that compile count as correct (Correct: 4 of 5). With no option: marks 10, 5, 10, 0 and 10, and 3 correct out of 5, which is what the trace says.

Now, the most interesting detail: after ./grade.sh submissions | tee report.txt, echo $? gives 0, even though the grader ended with 1. Why? Because in a pipeline, $? is the code of the last command, and the last one is tee, which saved the report without any trouble. As soon as we launch it without a pipe (> /dev/null), echo $? gives the real 1, and the && … || prints Needs review. Finally, wc -l report.txt counts 6 lines: 5 students + 1 summary.

Questions you might ask

Why shift instead of checking $2 when there’s an option? Because then the whole script would have to know whether there was an option (“is the folder in $1 or $2?”). With shift, from line 21 onwards the folder is always in $1 and $# must always be 1.

Why if ! compile …? The ! inverts the code: it goes into the then when compile fails. That way the odd case (doesn’t compile) is dealt with first and with continue, and the rest of the loop stays un-nested.

Why [ "$mode" = compile ] with = and not -eq? Because mode is a word. -eq only works for numbers; with words it would give integer expression expected, like seven in the theory.

Why does dani get a 5 and not a 0? It’s a grading decision: their program compiles and does something, just not the right thing. The script reflects that with a middle mark and also says what it got (got 12) so it can be checked by hand.

Why does the exit status compare correct with total? So the script can be used from another script or an if without reading the text: if ./grade.sh submissions > /dev/null; then echo "all good"; fi.

Try it yourself

• Add an -o file option that saves the report to that file (hint: the name comes after the option, so you’ll need shift 2).
• Make the mark of those that compile with warnings drop by one point (hint: save the errors to a file instead of /dev/null and check it with -s).

Common mistakes in this practice

• Quotes with the wildcard inside ("$folder/*.c"): the * isn’t expanded and the loop runs once with a name that doesn’t exist.
• Deciding by whether the executable exists: an old executable makes a submission that no longer compiles look fine. Use gcc‘s code and delete the old one with rm -f.
• Running a program without <: it waits for the keyboard and the script looks stuck.
• Checking $? too late or after a pipeline: what you see is another command’s code (the echo or the tee).
• Comparing words with -eq (or numbers with =): -eq, -lt… for numbers; =, != for strings.
• Using return in a function that has to stop the script (like usage): return only leaves the function, and the script would carry on with the wrong data.

Try it in your terminal

Two tools for when a script doesn’t do what you expect: bash -x, which shows each command before running it, and bash -n, which checks the syntax without running anything. Type this in the test folder:

$ cat > mini.sh << 'EOF'
> #!/bin/bash
> for f in submissions/ana.c submissions/pablo.c; do
>     gcc "$f" -o prog 2> /dev/null && echo "OK $f" || echo "ERROR $f"
> done
> EOF
$ bash -x mini.sh
+ for f in submissions/ana.c submissions/pablo.c
+ gcc submissions/ana.c -o prog
+ echo 'OK submissions/ana.c'
OK submissions/ana.c
+ for f in submissions/ana.c submissions/pablo.c
+ gcc submissions/pablo.c -o prog
+ echo 'ERROR submissions/pablo.c'
ERROR submissions/pablo.c
$ cat > broken.sh << 'EOF'
> #!/bin/bash
> if [ -f submissions/ana.c ]; then
>     echo "ana has submitted"
> EOF
$ bash -n broken.sh
broken.sh: line 4: syntax error: unexpected end of file
$ echo $?
2

What to watch for:

1. With bash -x, each command appears with a + in front and with the variables already substituted: you see gcc submissions/ana.c -o prog, not gcc "$f" -o prog.
2. In pablo’s round, right after gcc comes echo 'ERROR …': the && skipped the OK because gcc returned 1, and the || ran its part.
3. The + lines go to stderr: if you do bash -x mini.sh 2> trace.txt, the normal output appears on screen and the trace goes to the file.
4. bash -n broken.sh finds that the fi is missing (unexpected end of file, on line 4, which is where the file ends) and returns 2, without having run the echo. It’s the first thing worth doing with a new script.

Summary

• A serious script checks its parameters at the start and ends with exit 2 and a message to stderr if it’s called wrongly.
• Decisions are made with exit statuses (if gcc …, cmp -s, [ -s file ]), not by looking at what’s left in the folder; and old stuff is deleted before generating the new.
• A function that returns a number with return and a case $? that turns it into text keep the calculation apart from the presentation.
• shift always leaves the parameters in the same place after processing the options.
• The script has to return a code that means something (0 = all good), and watch out for $? after a pipeline.
• bash -n for the syntax and bash -x to see what it really does, line by line.

Checklist: have you practised it?

• Have you run the three programs and checked the outputs?
• Can you explain why pablo doesn’t appear in ls submissions even if it had compiled another day?
• Can you say what the run_test function returns for each student and why?
• Do you understand why echo $? gives 0 after | tee report.txt?
• Have you tried the “Try it yourself” tasks without looking at the theory?
• Have you used bash -x at least once to follow one of your scripts?

If you answered yes to almost everything, you’re ready for the shell scripts exercises, where I no longer give you the code. See you there!

Next step: exercisesShell scripts in Linux — exercises with parameters that contain spaces, grades that need validating and a list of students to mark (and the file called “3” that appeared in the folder)

Did this article help you? Here is how you can support SergioLearns →