Follow changes in your repositories

If you are a Linux (or UNIX) user and using SCM (Source Control Management) software like Fossil and Git, maybe like me you do not want to immediately commit your changes. You do not want drafts to be tracked. But from time to time you forget about those uncommitted changes and risk to lose your work.

Would you like to see a list of your repositories with uncommitted changes every time you open a new shell session or a terminal? To have something like this?

Uncommitted changes present in these directories:
/home/nales/git-project
/home/nales/fossil-project
/home/nales/website
~ $

This article is about a set of scripts I daily use to automatically follow the changes in my repositories. You can find my scripts (and others) in my scripts repository. Those scripts work on Linux, they may work on BSD systems or MacOS. The goal of this article is not to explain line-by-line the scripts, but to give you the idea of using a similar solution and to show you how to adapt my solution into yours.

The term SCM in this article refers to Source Control Management software like Fossil or Git. If you do not know what is an SCM, I highly recommend you to check out Git which is the most popular one right now in 2020 (even if I prefer Fossil). In a nutshell, an SCM is a software used to organize changes in your files (usually code source). You can diverge from the current state to commit changes and apply them later, you can collaborate with other people and merge their changes to yours, you can also check out the history of changes to see how your files evolved and revert changes.

Planning the solution

The goal is to check out if there are uncommitted changes from our side in the repositories we follow. The goal is not to check out distant changes from other people. We use Fossil and Git.

There will be two scripts for that. The first script, which-scm, is used to find which SCM is used in a directory and what is the root path of the repository. The second script, follow-scm, is used to add the repositories we want to follow and to check out uncommitted changes. When we open a new shell in a terminal, the repositories with uncommitted changes are displayed.

Finding the SCM with which-scm

Both Fossil and Git create an hidden file in the root path of their repositories. Fossil creates a SQLite database name .fslckout. (Fossil was created by the developers of SQLite.) Git creates a directory named .git.

We want which-scm to print in the standard output the name of the SCM and the root path of the repository separated by a tabulation to simplify integration in other scripts. If an SCM is found, the script returns 0. If no SCM are found, because we are not in a repository or in another SCM repository (SVN for example), nothing is printed and the script returns 1.

which-scm
#!/bin/sh

print_info_and_exit() {
    printf "%s\t%s\n" "$1" "$(pwd)"
    exit 0
}

check_local() {
    if [ -d ".git" ]; then
        print_info_and_exit "git"
    elif [ -f ".fslckout" ]; then
        print_info_and_exit "fossil"
    fi;
}

# If no arguments given, takes the current directory as an argument.
path="${1:-$(pwd)}"

if [ ! -d "$path" ]; then
    printf "%s is a not a directory path.\n" "$path" 1>&2
    exit 2
fi

# Checking the directory and its parents until giving up at "/" path.
cd "$path"
while [ "$(pwd)" != "/" ]; do
    check_local
    cd ..
done

exit 1

Using this script in a shell gives us those results.

$ which-scm /home/nales/git-project/src/
git	/home/nales/git-project
$ which-scm /home/nales/fossil-project/src/
fossil	/home/nales/fossil-project
$ which-scm /home/nales/not-a-repository/
$ echo $?
1

Writing follow-scm

follow-scm will be used to choose which repositories in the system we want to follow. It will use which-scm to keep track of the SCM used and the root repository path. Let's write it now.

Defining the files and script commands

We need to find where to store the list of repositories follow-scm will follow. In order to let the user organize where the files will be in its system, the script will use the XDG Base Directory specification. We write our data in the XDG_DATA_HOME directory in a follow-scm file.

There will be a bunch of commands to define. I prefer to use variables to store the commands, it makes refactoring easier. I will not implement in this article every command because this is not the goal. See my scripts repository for a full implementation. Here is the source code skeleton.

follow-scm
#!/bin/sh

COMMAND_HELP="help"
COMMAND_ADD="add"
COMMAND_REMOVE="remove"
COMMAND_AUTOREMOVE="autoremove"
COMMAND_CHECK="check"
COMMAND_LIST="list"
COMMAND_EDIT="edit"

DATA_FILE="${XDG_DATA_HOME:-$HOME/.local/share}/follow-scm"
if [ ! -e "$DATA_FILE" ]; then
    touch "$DATA_FILE"
fi

# We will need a temporary file for remove and autoremove commands
# WARNING: Linux uses /tmp but I am not sure for other OSes. Adapt the script!
TEMP_DIR="${TMPDIR:-/tmp}"
TEMP_FILE="$TEMP_DIR/follow-scm-$$"

is_known_scm() {
    [ "$1" = "fossil" ] ||
    [ "$1" = "git" ]
}

# We will define the commands here.

if [ $# -eq 0 ]; then
    help
else
    case "$1" in
        "$COMMAND_HELP")
            shift; help "$@"
            ;;
        "$COMMAND_ADD")
            shift; add "$@"
            ;;
        "$COMMAND_REMOVE")
            shift; remove "$@"
            ;;
        "$COMMAND_AUTOREMOVE")
            shift; autoremove
            ;;
        "$COMMAND_LIST")
            shift; list
            ;;
        "$COMMAND_EDIT")
            shift; edit
            ;;
        "$COMMAND_CHECK")
            shift; check
            ;;
        *)
            print_unknown_command "$1"
    esac
fi

Adding repositories to follow

The add takes a path as an argument. If there are no paths given, follow-scm uses the current directory. We simply store the result of which-scm in the data file.

follow-scm
add() {
    path="${1:-$(pwd)}"
    which_output="$(which-scm "$path")"
    if [ -z "$which_output" ]; then
        printf "%s is not in a SCM repository.\n" "$path"
        exit 1
    fi
    scm="$(printf %s "$which_output" | cut -f1)"
    if ! is_known_scm "$scm"; then
        # Should print error and instructions for help here
        exit 1
    fi

    # No need to add again an already present entry
    if grep --quiet "^$which_output\$" "$DATA_FILE"; then
        return
    fi
    printf "%s\n" "$which_output" >> "$DATA_FILE"
}

Then just add the repositories you want to follow.

$ follow-scm add /home/nales/git-project/src/
$ follow-scm add /home/nales/fossil-project/src/

Checking changes in Fossil and Git

With Fossil, we need two commands to check changes. The command fossil changes display changes in the already tracked files, the command fossil extras display new files that are not already tracked. If no changes are present, nothing is displayed.

With Git, we only need git status -s to display changes. If no changes are present, nothing is displayed.

The check command reads the data file filled by the add command, and check one by one each repository. It prints the repositories with uncommitted changes.

follow-scm
get_scm_from_entry() {
    printf "%s" "$1" | cut -f1
}

get_path_from_entry() {
    printf "%s" "$1" | cut -f2
}

check() {
    line_number=0
    while read -r line; do
        line_number=$((line_number + 1))

        # Here tests should be present to check the integrity of the entries
        # stored in the data file. To reduce the code, I skip them.

        path="$(get_path_from_entry "$line")"
        scm="$(get_scm_from_entry "$line")"

        cd "$path" || continue
        if [ "$scm" = "git" ] && [ -n "$(git status -s)" ]; then
            printf "%s\n" "$path"
        elif [ "$scm" = "fossil" ] && [ -n "$(fossil changes && fossil extras)" ]; then
            printf "%s\n" "$path"
        fi
    done < "$DATA_FILE"
}

Now we can check the repositories with uncommitted changes:

$ follow-scm check
/home/nales/fossil-project

Removing entries

Without going into details here, to remove some repositories in the data file you can use something like that:

follow-scm
remove() {
    path="${1:-$(pwd)}"
    which_output="$(which-scm "$path")"
    if [ -z "$which_output" ]; then
        # Print here the error and suggest to use autoremove command
        exit 1
    fi
    grep -v "^$which_output\$" "$DATA_FILE" > "$TEMP_FILE"
    mv -f "$TEMP_FILE" "$DATA_FILE"
}

autoremove() {
    printf "" > "$TEMP_FILE"
    while read -r line; do
        # ...
        # Test here each line, if the entry is incorrect: skip it.
        # ...

        # Only write correct entries
        printf "%s\n" "$line" >> "$TEMP_FILE"
    done < "$DATA_FILE"
    mv -f "$TEMP_FILE" "$DATA_FILE"
}

To remove the currently followed Git repository from the list of followed repositories, do:

$ follow-scm remove /home/nales/git-project/src/

I will not show other commands like help or list because it is not the goal here.

Displaying the list in new shell sessions

We want to display the repositories with uncommitted changes when opening a new session in a shell (when you open a terminal for example). To do that, add in your .bashrc or your .zshrc (or any equivalent file in your system) those lines:

# Print in red the list of followed repositories with uncommitted changes.
echo -en "\033[1;31m"
follow-scm check
echo -en "\033[0m"

It works, but this is not efficient. Every time you open a new shell session, you have to wait the end of the command before doing anything. Let's improve that.

The first thing we do is to create a script called follow-scm-check which will print the result of follow-scm check in a cache file, or delete this file if there are no results.

follow-scm-check
#!/bin/sh

# If called as a cron task, do not forget to export PATH environment variable
# so the path for this script and its dependencies (follow-scm, git, fossil...)
# are present. If you do not, maybe nothing will work because the shell used by
# cron cannot find the scripts.

CACHE_FILE="${XDG_CACHE_HOME:-$HOME/.cache}/follow-scm-check"

result="$(follow-scm check)"

if [ -n "$result" ]; then
    echo "Uncommitted changes present in these directories:" > "$CACHE_FILE"
    printf "%s\n" "$result" >> "$CACHE_FILE"
elif [ -f "$CACHE_FILE" ]; then
    rm "$CACHE_FILE"
fi

Now we will call this script every 10 minutes using a cron task. If you do not know what is a cron task, it is a task that is scheduled according to the rules you set. Here is the cron task in my system, use the crontab -e command to add it:

*/10 * * * * export PATH=$PATH":$HOME/bin:$HOME/.local/bin:/usr/local/bin/"; follow-scm-check

Frankly cron tasks are a real pain to work with. They do not execute into your usual shell so even if you script works when you execute it manually, maybe it does not work when executed as a cron task. You musk check the PATH environment configuration. You want to see the error logs when there is a problem? Silly you! Do you really think that on Linux there is a standard way across all different Linux flavors for this kind of simple things? No wonder why $CURRENT_YEAR is still not the year of the Linux desktop...

The explanation of the cron task above.

  1. We set up the task to be run every 10 minutes with */10 * * * *.
  2. The rest of the line is about the script to execute. The set up for PATH variable correspond of my needs on my system.
    • I saved follow-scm-check at $HOME/bin/ because it is a personal script.
    • I have my fossil executable at $HOME/.local/bin/ which is a standard directory to put executables.
    • And the scripts from my scripts repository which include fossil-scm are installed in /usr/local/bin/ as does suckless for their software. (You should check out the Makefile in my repository in case changes were made since I wrote this article.)
  3. Finally we can call follow-scm-check.

Now we can update the code in the .bashrc or .zshrc (or any equivalent) to print the uncommitted repositories.

# Print in red the list of followed repositories with uncommitted changes.
if [ -f ~/.cache/follow-scm-check ]; then
    echo -en "\033[1;31m"
    cat ~/.cache/follow-scm-check
    echo -en "\033[0m"
fi

Tip: following every repository in your system

Instead of manually searching and adding all the repositories to follow in your system, you can use find to automatize that.

To find the repositories, we can search for the .git directory and the .fslckout file. See the manual of find in the OPERATORS section for more information. We assume the repositories are in the directory Projects.

$ find Projects \( -type d -a -name '.git' -o -type f -a -name '.fslckout' \)
Projects/Website/.fslckout
Projects/Scripts/.git
Projects/G-Code-Viewer/.git

find has a command to execute a script for each thing it finds. We cannot use directly follow-scm add because it works only on directories paths but here find also returns .fslckout paths. The simplest solution is to create a temporary script.

add-scm
#!/bin/sh
# ${1%/*} removes the smallest suffix in "$1" with the pattern "/*". Here it
# removes from the argument "/.git" and "/.fslckout" so it returns the
# directory path.
# For example: "/home/nales/project/.git" becomes "/home/nales/project"
follow-scm add "${1%/*}"

We can now mix together our find command with our temporary script.

$ find Projects \( -type d -a -name '.git' -o -type f -a -name '.fslckout' \) -exec ./add-scm {} \;

All the repositories are now followed. If you want to remove some, you can use follow-scm edit which opens follow-scm data file in a text editor. Here is the code of this command if you are curious:

follow-scm
edit() {
    # Check VISUAL then EDITOR environment variable else choose vi
    editor="${VISUAL:-${EDITOR:-vi}}"
    "$editor" "$DATA_FILE"
}

In summary

To display all repositories in your system with uncommitted changes, you can use follow-scm present in my scripts repository or create your own version. After that, simply create a task cron to regularly make a check. Print the check result when you open a shell session.

With all the explanation I gave you above, you are more than ready to make your own version. Have fun!