Commandline Source Fixes Foo

From time to time I need to cleanup source trees. Today I needed to do that again, here are some command liners to get some work done.

If you’re on Windows, all you need to do to get these running is to install git for windows which has git bash and all the commands used in these examples.

Convert Line Endings / Line Separators

First of all it’s good to review if the find command-line actually finds the file looking for. E.g. to exclude some directories (here exemplary .git for git version control and .idea for Phpstorm and other Idea IDEs) and then list the file extensions that would be find:

find . \( -name '.git' -o -name '.idea' \) -prune -o \
    -type f -printf '%f\n' | awk -F . '{print $NF}' | sort -u

Example: List of file extensions

$ find . \( -name '.git' -o -name '.idea' \) -prune -o \
>     -type f -printf '%f\n' | awk -F . '{print $NF}' | sort -u
gitattributes
gitignore
json
md
php
xml
yml

This shows it’s save to operate on these. Lets ensure all line-endings are unix and not dos:

find . \( -name '.git' -o -name '.idea' \) -prune -o \
    -type f -exec dos2unix -bUvt {} \; 2>&1

This executes dos2unix (here in test-mode, remove -t switch to apply changes) on each file (redirecting stderr to stdout so it’s easier to grep or less). Dos2unix allows more conversions, use dos2unix --help more more info.

Something new I tried today was to apply such a command only onto files that have been touched by the last commit. In a clean staging area after that commit, I could apply dos2unix with the help of git diff-tree and a Bash loop:

git diff-tree --no-commit-id --name-only -r HEAD \
    | while read line ; do dos2unix -bUvt "$line" ; done

So instead of the find operation, I create a list of files to operate on with git and then a while read line ; do ; done loop invokes the command.

\ No newline at end of file

Another common change to apply is to add newlines at the end of files. Some background information about why a newline at the end of file is useful is given in Sanitizing files with no trailing newline (May 2010; by waldner). This one was not so easy for me to find as I wanted to invoke it again via find, but I finally made it working like a charm with the help of sed and the nice Gnu extension of -i (edit file in place) it has – to great extend because of How to add a newline to the end of a file? (Unix & Linux SE):

find . \( -name '.git' -o -name '.idea' \) -prune -o \
    -type f -exec sed -i -e '$a\' {} \;

this does not produce any output, but you can review the changes then with git diff. All those \ No newline at end of file should be gone then.

As usual, keep a backup before running modifications over a whole directory tree automated. Take care to not traverse into directories where you don’t want to.