Brian's Useful Utilities Set

Contents

Introduction to the utilities

I’ve been working with Unix and Linux since the late 1990s. In that time I’ve written a number of utility scripts that I find useful. The two dozen utilities persented here are listed in perceived order of usefulness (to me.)

You’ll notice some begin with the letter “j”. These are usually little scripts that enhance or alter the typical output of the Linux command with the name that follows the j. For example, jlsusb is an enhancement to the lsusb command that displays useful device information presented as a tree. I chose “j” because the least number of Linux commands start with that letter, and it’s easy to find on most QWERTY/AZERTY/QZERTY keyboards.

All these programs are copyright by me. They are posted here under the terms of the GNU Public License version 2 or any later version.

If you find and correct bugs, or improve any of these programs, I’d love to get a copy of the updated code. If I like the changes I’ll post the updated version here with a note of who contributed the change.


jgrep: pretty-print output from a recursive grep

Of all the utilities I’ve wiritten, this is my personal favourite. I use it many times a day when searching for information or looking at a code base.

With only one parameter (an extended regular expression), jgrep performs a recursive search through the current directory and all its subdirectories for matches. Results are displayed with a line naming the file where matches were found followed by the matches prefixed with line numbers.

Parameters beyond the first are the names of files and/or subdirectries to search.

Here are a few examples.

Example 1: Find all the contexts in /etc/asterisk

[asterisk@pbx ~]$ jgrep '^\[' /etc/asterisk | less
/etc/asterisk/ari_additional.conf:
    10: [freepbxuser]

/etc/asterisk/dnsmgr.conf:
     1: [general]

/etc/asterisk/amd.conf:
    10: [general]

/etc/asterisk/asterisk.conf:
     1: [directories]
     9: [options]
    14: [files]

/etc/asterisk/meetme.conf:
     1: [general]
     4: [rooms]

/etc/asterisk/cdr_adaptive_odbc.conf:
     1: [asteriskcdrdb]

/etc/asterisk/pjsip.auth.conf:
    11: [312-auth]
    17: [316-auth]
    23: [306-auth]
    29: [303-auth]
    35: [305-auth]
    41: [313-auth]
    47: [338-auth]
    53: [337-auth]
    59: [Trunk-A]
    65: [Trunk-B]
--less--

Example 2: Search the kernel documentation for “Real-time” or “realtime”

[brian@sparrow ~]$ cd /var/tmp/linux-4.16.9/Documentation
[brian@sparrow Documentation]$ jgrep '[Rr]eal-?[Tt]ime' | less
ABI/testing/rtc-cdev:
     6:                 The ioctl interface to drivers for real-time clocks (RTCs).

ABI/testing/sysfs-devices-system-ibm-rtl:
     6:                 out of Premium Real-Time Mode (PRTM), as well as the

ABI/testing/sysfs-power:
   127:                 CAUTION: Using it will cause your machine's real-time (CMOS)

admin-guide/devices.txt:
  2224:  150 char       Real-Time Linux FIFOs

admin-guide/kernel-parameters.txt:
  2682:                         real-time systems.
  3360:                         real-time workloads.  It can also improve energy
  3368:                         This improves the real-time response for the
  3437:                         the default is zero (non-realtime operation).
  3656:                         real-time latency, and degrade energy efficiency.
  3663:                         real-time latency, CPU utilization, and

block/bfq-iosched.txt:
    84: Low latency for soft real-time applications
    86: Also soft real-time applications, such as audio and video
   128: . real-time recording of data in live-dumping applications (e.g.,
   190:     real-time applications (e.g., video or audio players/streamers),
   197:     interactive applications, and a stronger form for soft real-time
   303: weight of the queues associated with interactive and soft real-time
   419: real-time applications are privileged and experience a lower latency,
   546: weight of the queues associated with interactive and soft real-time

block/ioprio.txt:
    20: IOPRIO_CLASS_RT: This is the realtime io class. This scheduling class is given
--less--

Example 3: List classes and methods in a python file

[brian@sparrow ~]$ cd /usr/lib/python3.7/site-packages/markdown/extensions
[brian@sparrow extensions]$ jgrep '^[[:space:]]*(class|def)\b' toc.py

toc.py:
    25: def slugify(value, separator):
    35: def unique(id, ids):
    47: def stashedHTML2text(text, md):
    49:     def _html_sub(m):
    61: def nest_toc_tokens(toc_list):
   124: class TocTreeprocessor(Treeprocessor):
   125:     def __init__(self, md, config):
   144:     def iterparent(self, node):
   156:     def replace_marker(self, root, elem):
   172:     def set_level(self, elem):
   179:     def add_anchor(self, c, elem_id):  # @ReservedAssignment
   191:     def add_permalink(self, c, elem_id):
   201:     def build_toc_div(self, toc_list):
   212:         def build_etree_ul(toc_list, parent):
   231:     def run(self, doc):
   279: class TocExtension(Extension):
   283:     def __init__(self, **kwargs):
   313:     def extendMarkdown(self, md):
   325:     def reset(self):
   330: def makeExtension(**kwargs):  # pragma: no cover

Code: jgrep


jman: format info or man page with table of contents

jman formats an info page (or man page if no info page is found), builds a table of contents for it, writes the result to a file in /tmp/jman/, then starts a text viewer to display the result.

For large pages such as bash or perlfunc, the resulting text file can be searched using jgrep.

Example 1: jman help text

[brian@sparrow ~]$ jman -h
usage: jman [-l spaces] [-w output_width] [-m] [-f man_file]

Parameters:
 -l spaces
      Number of leading spaces to use for indexing. Default 7.
 -w output_width
      Output width. Default 72.
 -m 
      Generate output based only on the man page and ignore any info page
 -f file
      Generate output from file instead of searching the man pages

JMAN_VIEWER environment variable:
* If unset, defaults to the value of $EDITOR, $PAGER, or 'less', in that
  order
* If set to 'vi' or 'vim' (directly or via $EDITOR), uses 'view' so that
  the page file is read-only
* If set to 'save_to_file', saves the formatted page as a file in the
  current directory

Example 2: “jman date”

The following partial display is the output from jman date:

DATE(1)                      User Commands                     DATE(1)

NAME
       date - print or set the system date and time

CONTENTS
        Line  Section
         18  SYNOPSIS
         22  DESCRIPTION
        189  EXAMPLES
        204  DATE STRING
        214  ENVIRONMENT
        219  AUTHOR
        222  REPORTING BUGS
        228  COPYRIGHT
        235  SEE ALSO

SYNOPSIS
       date [OPTION]... [+FORMAT]
       date [-u|--utc|--universal] [MMDDhhmm[[CC]YY][.ss]]

DESCRIPTION
       Display the current time in the given FORMAT, or set the system
       date.

       Mandatory arguments to long options  are  mandatory  for  short
       options too.
... remainder of page here ...

Example 3: “jman gawk”

Running jman gawk collects the entire set of info pages for the GNU awk program. Here’s the first part of the output:

GAWK Documentation
******************

Table of Contents
=================

    457. General Introduction

    1137. Foreword to the Third Edition

    1223. Foreword to the Fourth Edition

    1258. Preface
    1341.   History of 'awk' and 'gawk'
    1391.   A Rose by Any Other Name
    1421.   Using This Book
    1571.   Typographical Conventions
    1603.     Dark Corners
    1624.   The GNU Project and This Book
    1687.   How to Contribute
    1710.   Acknowledgments

    1816. 1 Getting Started with 'awk'
    1867.   1.1 How to Run 'awk' Programs
    1898.     1.1.1 One-Shot Throwaway 'awk' Programs
    1926.     1.1.2 Running 'awk' Without Input Files
    1975.     1.1.3 Running Long Programs
    2012.     1.1.4 Executable 'awk' Programs
    2080.     1.1.5 Comments in 'awk' Programs
    2132.     1.1.6 Shell Quoting Issues
    2320.   1.2 Data files for the Examples
    2379.   1.3 Some Simple Examples
    2497.   1.4 An Example with Two Rules
    2540.   1.5 A More Complex Example
 ... remainder of page here (approximately 36,400 lines) ...

Code: jman


vip: edit a file while preserving its modification time

I’ll admit to mild obsessive-compulsive behaviour regarding modification times (mtime, displayed by ls and file management programs) on files and directories. Once I consider a file complete, I don’t like having the mtime change if I have to go back hours, days, or months later to make a minor edit such as correcting a typo or fixing a trivial bug.

vip (“VI/Preserve”) edits one or more files using vim while preserving their modification timestamps and inode number.

The modification time is not perfectly preserved; it is incremented by one second so rsync and similar utilities that examine timestamps will know the file has actually changed.

Code: vip


save: make a copy of a file prior to changing it

The save command makes a copy of a file to filename.SAVE.ext. Typing save filename.ext is faster than typing the full command cp -a filename.text filename.SAVE.text.

Example

[brian@sparrrow kernel]$ save asn1_compiler.c

The above creates a new file named asn1_compiler.SAVE.c. You can then make potentially destructive changes to asn1_compiler.c without disrupting the original file.

There is no unsave program. To restore the original file, move the .SAVE file to the original name:

[brian@sparrrow kernel]$ mv -f asn1_compiler.SAVE.c asn1_compiler.c

Notes

Files that don’t have a “dot” in their name simply get .SAVE appended to the name.

A word of caution regarding configuration files. Many packages put configuration files into a directory and have a directive to include all .conf files in that directory. Using save in this situation can cause the program to read the configuration file twice: once by its actual .conf name and again by its .SAVE.conf name. Some packages are more tolerant of this situation than others. Some may give an error about duplicate directives. Others might read your modified .conf first and then the original .SAVE.conf file, effectively ignoring the changes you made.

Code: save


mvmp3: bulk rename files

Comment in the code:

Assist with the renaming of MP3 files, which typically have very long file names with embedded spaces.

Since that comment was written, the program has morphed into a bulk file rename utility. To use, simply type:

[brian@sparrow somedir]$ mvmp3 pattern

where pattern is either a single file name or a fileame pattern supported by you shell, such as *.jpg or file-????.txt. You can specify one or more patterns on the command line. The program puts the resulting file names into a list and loads it into the text editor named by the $EDITOR environment variable.

For example, here’s a possible rename file created by mvmp3 *.txt:

mt.txt                       [1]
notebooks.txt                [2]
sun-picture.txt              [3]
whitelist.txt                [4]
x1.txt                       [5]
x.txt                        [6]

Make whatever changes you desire, or no changes at all. Pretty much any character, including spaces but excluding “/”, can be used in the file name. Leave the [#] part at the end of the line unchanged, because that’s what the script uses to match up the new name you’ve given the file with the original.

In this example, we’ll use the file being edited in the text editor to rename “x.txt” to “Movie List 1.txt” and “x1.txt” to “Movie List 2.txt”:

mt.txt                       [1]
notebooks.txt                [2]
sun-picture.txt              [3]
whitelist.txt                [4]
Movie List 2.txt                       [5]
Movie List 1.txt                        [6]

When you exit the text editor, the script will rename the files:

* Not renaming mt.txt
* Not renaming notebooks.txt
* Not renaming sun-picture.txt
* Not renaming whitelist.txt
* Rename "x1.txt" to "Movie List 2.txt"
* Rename "x.txt" to "Movie List 1.txt"

Code: mvmp3


psre: list or send a signal to processes matching a regular expression

Given an extended regular expression, psre lists the processes whose lines in a standard ps -ef command match the expression. As such, the expression can match a user, a process name or number, a parameter used on the command line that started the command, or various combinations thereof.

Passing -k to psre causes a SIGTERM to be sent to the matched processes. An arbitrary signal name such as -HUP or a number such as -9 can be used as well.

Example: forcefully kill all processes running as user rogue:

[root@sparrow ~]# psre -9 ^rogue

Code: psre


sc: “systemctl” command shortener

sc implements a set of shortcuts for the systemctl command (and one for journalctl) using symlinks in /usr/local/bin:

Symlink Runs
sc systemctl
scs systemctl start (‘s’ = ‘start)
scr systemctl restart (‘l’ ‘ restart)
sck systemctl stop (‘k’ = ‘kill’)
scl systemctl reload (‘l’ = reload)
scq systemctl status (‘q’ = ‘query’)
sct systemctl status (‘t’ = ‘status’)
scj journalctl (scj NAME runs journalctl --catalog --unit=NAME)
sch (provides help for the sc command)

Setup:

  1. Put the sc script into /usr/local/bin
  2. Run the sc --setup to set up the symlinks

Code: sc


genTOC: Generate a Table of Contents for a text file

Generates a nested table of contents for a file, based on the formatting model I initially saw in the MySQL documentation.

The script looks for lines that look like the following:

Some text
=========

Next heading level
------------------

    Another heading level
    ~~~~~~~~~~~~~~~~~~~~~

Note the underline, which consists of repeating non-alphanumeric characters of the same length as the text on the line above. Levels are assigned automatically as each new underline is encountered.

The above might be formatted in the Table of Contents as:

Table of Contents
=================
 8  Some text
11    Next heading level
14      Another heading level

The table of contents is inserted after the line ‘Table of Contents’ (and its underline) is encountered. Any existing table of contents is replaced.

Code: genTOC


setext-headings: promote or demote setext-style heading characters

A lot of the files I write (including the source files for this blog) use the following “underline” characters for headings:

  • Level 1: **********
  • Level 2: ==========
  • Level 3: ----------
  • Level 4: ~~~~~~~~~~
  • Level 5: ..........
  • Level 6: ''''''''''

On more than one occasion, usually when transferring information from a file I had written earlier into the blog, the heading levels were wrong. Typically I had to replace level 1 headings with level 2, level 2 headings with level 3, and so on (I refer to this a ‘demoting’ the headings.) This can be tedious, so I wrote setext-headings to do the work for me.

The program is intended to be symlinked to setext-headings-promote and setext-headings-demote; it can’t be run simply as setext-headings.

Code: setext-headings


jdf: pretty-print output from “df”

Presents nicely formatted output from ‘df’. Also aims to set a record of some sort for the greatest ratio of comments to lines of code 😊

The script also filters out tmpfs filesystems.

Example: Normal df output

[brian@sparrow]$ df -h    # 
Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        5.8G     0  5.8G   0% /dev
tmpfs                           5.9G  348M  5.5G   6% /dev/shm
tmpfs                           5.9G  1.9M  5.9G   1% /run
/dev/mapper/vg_sparrow-lv_root   20G  8.8G  9.8G  48% /
tmpfs                           5.9G  151M  5.7G   3% /tmp
tmpfs                           5.9G  235M  5.6G   4% /r
/dev/sda2                       976M  229M  680M  26% /boot
/dev/sda1                       599M   20M  580M   4% /boot/efi
/dev/mapper/vg_sparrow-lv_home   63G   30G   30G  50% /home
/dev/mapper/vg_sparrow-lv_var    79G   51G   24G  68% /var
tmpfs                           1.2G  156K  1.2G   1% /run/user/1000
tmpfs                           5.9G  378M  5.5G   7% /home/yves
tmpfs                           5.9G  411M  5.5G   7% /home/zero
tmpfs                           1.2G     0  1.2G   0% /run/user/0
/dev/dm-7                       701G  410G  292G  59% /r/npv
/dev/sdb1                       6.1G  4.1G  2.1G  67% /run/media/brian/GE9-NTFS
/dev/sdb2                       1.2G   61M  1.2G   6% /run/media/brian/GE9-FAT32

jdf output:

[brian@sparrow]$ jdf
/dev/mapper/vg_sparrow-lv_root  20G   8.8G  9.8G  48%  /
/dev/sda2                       976M  229M  680M  26%  /boot
/dev/sda1                       599M  20M   580M  4%   /boot/efi
/dev/mapper/vg_sparrow-lv_home  63G   30G   30G   50%  /home
/dev/mapper/vg_sparrow-lv_var   79G   51G   24G   68%  /var
/dev/dm-7                       701G  410G  292G  59%  /r/npv
/dev/sdb1                       6.1G  4.1G  2.1G  67%  /run/media/brian/GE9-NTFS
/dev/sdb2                       1.2G  61M   1.2G  6%   /run/media/brian/GE9-FAT32

Code: jdf


jlsusb: pretty-print output from ‘lsusb’

As as January 2020, the output from lsusb is helpful but not very useful. You have a choice between two primary formats.

“lsusb” format 1: A simple listing of devices, with device IDs and descriptions:

[brian@sparrow ~]$ lsusb
Bus 001 Device 002: ID 8087:8001 Intel Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 003: ID 0bc2:2320 Seagate RSS LLC USB 3.0 bridge [Portable Expansion Drive]
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 006: ID 0bda:0129 Realtek Semiconductor Corp. RTS5129 Card Reader Controller
Bus 002 Device 027: ID 8087:07dc Intel Corp. 
Bus 002 Device 003: ID 064e:920b Suyin Corp. Integrated_Webcam_HD
Bus 002 Device 022: ID 045e:00a4 Microsoft Corp. Compact Optical Mouse, model 1016
Bus 002 Device 029: ID 0930:6545 Toshiba Corp. Kingston DataTraveler 102/2.0 / HEMA Flash Drive 2 GB / PNY Attache 4GB Stick
Bus 002 Device 024: ID 04b4:0060 Cypress Semiconductor Corp. Wireless optical mouse
Bus 002 Device 019: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 002 Device 016: ID 05e3:0606 Genesys Logic, Inc. USB 2.0 Hub / D-Link DUB-H4 USB 2.0 Hub
Bus 002 Device 015: ID 072f:90cc Advanced Card Systems, Ltd ACR38 SmartCard Reader
Bus 002 Device 014: ID 1a40:0101 Terminus Technology Inc. Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

“lsusb” format 2: USB hub/device tree with technical inforamation (device and interface IDs; class, driver, and speed) but no device IDs or descriptions:

  [brian@sparrow ~]$ lsusb --tree
  /:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
      |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=uas, 5000M
  /:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/11p, 480M
      |__ Port 3: Dev 14, If 0, Class=Hub, Driver=hub/4p, 480M
          |__ Port 1: Dev 15, If 0, Class=Chip/SmartCard, Driver=usbfs, 12M
          |__ Port 2: Dev 16, If 0, Class=Hub, Driver=hub/4p, 12M
              |__ Port 3: Dev 19, If 2, Class=Human Interface Device, Driver=usbhid, 12M
              |__ Port 3: Dev 19, If 0, Class=Human Interface Device, Driver=usbhid, 12M
              |__ Port 3: Dev 19, If 1, Class=Human Interface Device, Driver=usbhid, 12M
              |__ Port 4: Dev 24, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
              |__ Port 4: Dev 24, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
          |__ Port 3: Dev 29, If 0, Class=Mass Storage, Driver=usb-storage, 480M
          |__ Port 4: Dev 22, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
      |__ Port 5: Dev 3, If 0, Class=Video, Driver=uvcvideo, 480M
      |__ Port 5: Dev 3, If 1, Class=Video, Driver=uvcvideo, 480M
      |__ Port 6: Dev 27, If 0, Class=Wireless, Driver=btusb, 12M
      |__ Port 6: Dev 27, If 1, Class=Wireless, Driver=btusb, 12M
      |__ Port 8: Dev 6, If 0, Class=Vendor Specific Class, Driver=rtsx_usb, 480M
  /:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
      |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M

jlsusb combines the two, displaying the hub/device tree along with device desciptions and speeds.

  [brian@sparrow ~]$ jlsusb
  Root Hub 1: 2.0 root hub (Linux Foundation), 480 Mb/s
  └─ Port 1: unknown device (Intel Corp), 480 Mb/s
  Root Hub 2: 2.0 root hub (Linux Foundation), 480 Mb/s
  ├─ Port 3: Hub (Terminus Technology Inc), 480 Mb/s
  │  ├─ Port 1: ACR38 SmartCard Reader (Advanced Card Systems, Ltd), 12 Mb/s
  │  ├─ Port 2: USB 2.0 Hub / D-Link DUB-H4 USB 2.0 Hub (Genesys Logic, Inc), 12 Mb/s
  │  │  ├─ Port 3: Wireless Mouse (ARESON), 12 Mb/s
  │  │  └─ Port 4: Unifying Receiver (Logitech, Inc), 12 Mb/s
  │  ├─ Port 3: Kingston DataTraveler 102/2.0 / HEMA Flash Drive 2 GB / PNY Attache 4GB Stick (Toshiba Corp), 480 Mb/s
  │  └─ Port 4: Compact Optical Mouse, model 1016 (Microsoft Corp), 1.5 Mb/s
  ├─ Port 5: Integrated_Webcam_HD (Suyin Corp), 480 Mb/s
  ├─ Port 6: unknown device (Intel Corp), 12 Mb/s
  └─ Port 8: RTS5129 Card Reader Controller (Realtek Semiconductor Corp), 480 Mb/s
  Root Hub 3: 3.0 root hub (Linux Foundation), 5000 Mb/s
  └─ Port 1: USB 3.0 bridge [Portable Expansion Drive] (Seagate RSS LLC), 5000 Mb/s

Device information is derived from two sources:

  1. Information reported by the device itself
  2. Information found in the usb.ids file

Vendor or device information may be missing fom either source. jlsusb examines both sources for each of the vendor and the device data and displays the better of the two.

The command jlusb -v gives a more complete but much longer display:

  [brian@sparrow ~]$ jlsusb -v
  * Reading USB ID file /usr/share/hwdata/usb.ids
    > File was last updated on 2020-01-09
    > Loaded 3014 vendors (2940 unique) and 17088 devices
    > 13950 unique device descriptions
    > Vendor with the most devices is Canon, Inc. with 650 devices
  > Device class IDs are in '(##/##/##)' format: Class/Subclass/Protocol
  Root Hub 1: 1d6b:0002, dev 1, 480 Mb/s
  │ Class = Hub/Unused/Full speed (or root) hub (09/00/00)
  │ (Device)  Manf="Linux" Product="EHCI Host Controller"
  │ (usb.ids) Manf="Linux Foundation" Product="2.0 root hub"
  └─ Port 1: 8087:8001, dev 2, driver=hub, 480 Mb/s
       Class = Hub/Unused/Single TT (09/00/01)
       (Device)  Manf="(No manufacturer string)" Product="(No product string)"
       (usb.ids) Manf="Intel Corp." Product="(No description available)"
  Root Hub 2: 1d6b:0002, dev 1, 480 Mb/s
  │ Class = Hub/Unused/Single TT (09/00/01)
  │ (Device)  Manf="Linux" Product="xHCI Host Controller"
  │ (usb.ids) Manf="Linux Foundation" Product="2.0 root hub"
  ├─ Port 3: 1a40:0101, dev 41, driver=hub, 480 Mb/s
  │  │ Class = Hub/Unused/Single TT (09/00/01)
  │  │ (Device)  Manf="(No manufacturer string)" Product="USB 2.0 Hub"
  │  │ (usb.ids) Manf="Terminus Technology Inc." Product="Hub"
  │  ├─ Port 1: 072f:90cc, dev 42, driver=usbfs, 12 Mb/s
  │  │    Class = Chip/SmartCard (0b/00/00)
  │  │    (Device)  Manf="ACS" Product="CCID USB Reader"
  │  │    (usb.ids) Manf="Advanced Card Systems, Ltd" Product="ACR38 SmartCard Reader"
  │  ├─ Port 2: 05e3:0606, dev 43, driver=hub, 12 Mb/s
  │  │  │ Class = Hub/Unused/Full speed (or root) hub (09/00/00)
  │  │  │ (Device)  Manf="ALCOR" Product="USB Hub 2.0"
  │  │  │ (usb.ids) Manf="Genesys Logic, Inc." Product="USB 2.0 Hub / D-Link DUB-H4 USB 2.0 Hub"
  │  │  ├─ Port 3: 25a7:2410, dev 46, driver=usbhid, 12 Mb/s
  │  │  │    Class = Human Interface Device/Boot Interface Subclass/Mouse (03/01/02)
  │  │  │    (Device)  Manf="ARESON" Product="Wireless Mouse"
  │  │  │    (usb.ids) Manf="(Vendor not found in database)" Product="(No description available)"
  │  │  └─ Port 4: 046d:c52b, dev 47, driver=usbhid, 12 Mb/s
  │  │       Class = Human Interface Device/Boot Interface Subclass/Keyboard (03/01/01)
  │  │       (Device)  Manf="Logitech" Product="USB Receiver"
  │  │       (usb.ids) Manf="Logitech, Inc." Product="Unifying Receiver"
  │  ├─ Port 3: 0930:6545, dev 44, driver=usb-storage, 480 Mb/s
  │  │    Class = Mass Storage/SCSI/Bulk-Only (08/06/50)
  │  │    (Device)  Manf="Kingston" Product="DataTraveler 2.0"
  │  │    (usb.ids) Manf="Toshiba Corp." Product="Kingston DataTraveler 102/2.0 / HEMA Flash Drive 2 GB / PNY Attache 4GB Stick"
  │  └─ Port 4: 045e:00a4, dev 45, driver=usbhid, 1.5 Mb/s
  │       Class = Human Interface Device/Boot Interface Subclass/Mouse (03/01/02)
  │       (Device)  Manf="Microsoft" Product="Microsoft(R) Compact Optical Mouse"
  │       (usb.ids) Manf="Microsoft Corp." Product="Compact Optical Mouse, model 1016"
  ├─ Port 5: 064e:920b, dev 3, driver=uvcvideo, 480 Mb/s
  │    Class = Miscellaneous Device (ef/00/01)
  │    (Device)  Manf="SuYin" Product="Integrated_Webcam_HD"
  │    (usb.ids) Manf="Suyin Corp." Product="(No description available)"
  ├─ Port 6: 8087:07dc, dev 5, driver=btusb, 12 Mb/s
  │    Class = Wireless (e0/00/01)
  │    (Device)  Manf="(No manufacturer string)" Product="(No product string)"
  │    (usb.ids) Manf="Intel Corp." Product="(No description available)"
  └─ Port 8: 0bda:0129, dev 7, driver=rtsx_usb, 480 Mb/s
       Class = Vendor Specific Class (ff/00/ff)
       (Device)  Manf="Generic" Product="USB2.0-CRW"
       (usb.ids) Manf="Realtek Semiconductor Corp." Product="RTS5129 Card Reader Controller"
  Root Hub 3: 1d6b:0003, dev 1, 5000 Mb/s
  │ Class = Hub/Unused (09/00/03)
  │ (Device)  Manf="Linux" Product="xHCI Host Controller"
  │ (usb.ids) Manf="Linux Foundation" Product="3.0 root hub"
  └─ Port 1: 0bc2:2320, dev 7, driver=uas, 5000 Mb/s
       Class = Mass Storage/SCSI (08/06/62)
       (Device)  Manf="Seagate" Product="Expansion"
       (usb.ids) Manf="Seagate RSS LLC" Product="USB 3.0 bridge [Portable Expansion Drive]"

Code: jlsusb


estimate-tar-size: estimate the size of a directory for ‘tar’

Given the ouptut of an “ls -la” or “ls -laR” command, computes the number of 512 byte blocks the directories and files in the listing would take when packed into a tar file. Output is to stdout in the form ‘#k’, (where ‘#’ is the expected size of the tar file in KiB,) which can be passed to ‘pv’ to monitor the progress of creating the tar file.

Example:

[brian@sparrow]$ cd /var
[brian@sparrow]$ DIR=www
[brian@sparrow]$ TAR_SIZE=$(ls -laR $DIR | estimate-tar-size); echo $TAR_SIZE
19323k
[brian@sparrow]$ tar cf - $DIR | pv -s$TAR_SIZE | bzip2 >/var/tmp/$DIR.tar.bz2
18.2MiB 0:00:03 [4.66MiB/s] [======================================================>] 100%
[brian@sparrow]$ 

Code: estimate-tar-size


jtouchdir: set the mtime on a directory based on its contents

Sets the mtime on a directory based on its contents.

For each directory name given, gets a lists of files in the directory, orders them according to extension, determines the most common extension, determines the timestamp of the most recently modified file of all the files that share that extension, then sets the modification timestamp of the directory to that time.

For example, running jtouchdir on a directory containing mostly JPEG files, the timestamp on the directory itself will be set to the same value as the most recent JPEG file in the directory.

Running jtouchdir -r recurses through the directory tree, setting the mtime on all the directories based on the method described above.

I wrote this program because I like the date and time on directories to reflect their contents as opposed to the last time a file or subdirectory in that directory was created or deleted.

Code: jtouchdir


find-icons.sh: Find icons whose names match a regular expression

Creates a directory named /tmp/icons-username (referred to as $ICONS_DN,) then locates files in directories whose path includes ‘icon’ or ‘icons’ and creates symlinks for them in $ICONS_DN/##x## (e.g $ICONS_DN/64x64) or $ICONS_DN/unspecified, and another symlink in $ICONS_DN/all.

The script expects an extended regular expression describing icons to find. For example, '\bedit\b' will find ‘edit’ but not ‘editor’ or ‘credit’,) Without a parameter it will find every icon it can on the system after confirming with the user.

When the search is complete and the directories are created, you can navigate to them using a GUI-based file manager, which should give you a nice display of the icons. Here is the all directory created by running the following command to find all the icons on sparrow that have editor in their name:

[brian@sparrow ~]$ find-icons.sh '\beditor\b'

image: find-icons

Code: find-icons.sh


inotifydev: run a script when a hot pluggable device is added to the system

Using inotifywait, watches /dev for device files being created and removed, usually in response to a USB device being connected and disconnected. When a device appears or disappears, this script searches the directory in which it was started for a script named inotifydev-X-device, where X is the name of the device that appeared (e.g. inotifydev-ttyUSB0-device for a serial device.) If no file with that name is present, it searches for a generic script by trimming the last character off the name (e.g. inotifydev-ttyUSB-device.) The search continues until it gets a match or only one charcter remains; that is it will search down to inotifydev-tt-device but not to inotifydev-t-device.)

If a matching script name is found, it is called with two parameters: CREATE or DELETE and the device name (without the leading /dev/).

inotifydev is intended to be run at startup. For systemd setups, put the following text into /etc/systemd/system/inotifydev.service, then run systemctl enable inotifydev (or run inotifydev --install to have this script to the work for you):

[Unit]
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/inotifydev start
ExecStop=/usr/local/bin/inotifydev stop
Restart=on-failure

[Install]
WantedBy=multi-user.target

The inotifydev --install function can also set up a System V init script on systems descended from Red Hat or Debian.

Use case: I use this script on my home server to watch for a USB backup drive being connected to the computer. When a drive is detected, it does a couple of checks to see if it’s a backup drive, and if it is launches my backup script.

Code: inotifydev


inotifydev-sd-device: USB drive handler for inotifydev

This is sample code for handling a USB drive using inotifydev. This might be useful on systems that aren’t running a GUI environment that automatically mounts drives when they’re connected.

When mounting a drive, this script attempts to assign the device and the mount point to a logged in user. It determines the user by listing all non-root processes, determining their owners, then searching the ‘passwd’ for file users that don’t have ‘nologin’ as their login shell.

Code: inotifydev-sd-device


ishostavail: quickly determine if a host is online

ishostavail is a TCP script to determine quickly if a given host is online by attempting to open a socket to it. If the open is successful, the script assumes the host was reachable and returns 0, else it returns 1.

By default the script attempts to open port 80 (http) on the assumption you’re trying to determine if a WWW host is available. You can over-ride the default port by specifying its name or number after the host.

The -v switch enables verbose mode; otherwise this script produces no output.

The -t (test) switch causes 0 to be returned on POSIX ECONNREFUSED, on the assumption you are testing to see if the host even exists, regardless of whether or not you can connect to a specified port.

Code: ishostavail


jiptables-L: pretty-print output from “iptables -L”

As of January 2020, “iptables -L” (display IP firewall tables) assumes the table names are fairly short. If they’re wider than 8 characters, the reamining items on the line are shiifted over by several spaces, making the output difficult to read. jiptables-L corrects this, as well as automatically including the -v (verbose: includes packet and byte counts, and the in/out interfaces) and --line-numbers switches in the iptables call.

By default the filter table is displayed. Other tables can be shown by naming them on the command line, and the all parameter shows all tables.

Example: iptables -L by itself

[root@sparrow ~]# iptables -L | less
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED,DNAT
ACCEPT     all  --  anywhere             anywhere            
INPUT_direct  all  --  anywhere             anywhere            
INPUT_ZONES  all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere             ctstate INVALID
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED,DNAT
ACCEPT     all  --  anywhere             anywhere            
FORWARD_direct  all  --  anywhere             anywhere            
FORWARD_IN_ZONES  all  --  anywhere             anywhere            
FORWARD_OUT_ZONES  all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere             ctstate INVALID
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
OUTPUT_direct  all  --  anywhere             anywhere            

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination         
FWDI_public  all  --  anywhere             anywhere            [goto] 
FWDI_public  all  --  anywhere             anywhere            [goto] 
--less--

jiptables-L output:

[root@sparrow ~]# jiptables-L | less

----- filter table -----

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num  pkts   bytes  target             prot  opt  in      out     source    destination  additional                                                                                         
1    13M    5705M  ACCEPT             all   --   any     any     anywhere  anywhere     ctstate RELATED,ESTABLISHED,DNAT 
2    3723   223K   ACCEPT             all   --   lo      any     anywhere  anywhere      
3    303K   60M    INPUT_direct       all   --   any     any     anywhere  anywhere      
4    303K   60M    INPUT_ZONES        all   --   any     any     anywhere  anywhere      
5    348    19923  DROP               all   --   any     any     anywhere  anywhere     ctstate INVALID 
6    298K   59M    REJECT             all   --   any     any     anywhere  anywhere     reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num  pkts   bytes  target             prot  opt  in      out     source    destination  additional                                                                                         
1    0      0      ACCEPT             all   --   any     any     anywhere  anywhere     ctstate RELATED,ESTABLISHED,DNAT 
2    0      0      ACCEPT             all   --   lo      any     anywhere  anywhere      
3    0      0      FORWARD_direct     all   --   any     any     anywhere  anywhere      
4    0      0      FORWARD_IN_ZONES   all   --   any     any     anywhere  anywhere      
5    0      0      FORWARD_OUT_ZONES  all   --   any     any     anywhere  anywhere      
6    0      0      DROP               all   --   any     any     anywhere  anywhere     ctstate INVALID 
7    0      0      REJECT             all   --   any     any     anywhere  anywhere     reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT 4511K packets, 25G bytes)
num  pkts   bytes  target             prot  opt  in      out     source    destination  additional                                                                                         
1    13435  13M    ACCEPT             all   --   any     lo      anywhere  anywhere      
2    4511K  25G    OUTPUT_direct      all   --   any     any     anywhere  anywhere      

Chain FORWARD_IN_ZONES (1 references)
num  pkts   bytes  target             prot  opt  in      out     source    destination  additional                                                                                         
1    0      0      FWDI_public        all   --   enp7s0  any     anywhere  anywhere     [goto] 
2    0      0      FWDI_public        all   --   +       any     anywhere  anywhere     [goto] 
--less--

Code: jiptables-L


set-fg-colour-vars: set up envronment variables with colour names

set-fg-colour-vars sets up the following environment variables:

FG_WHITE
FG_GRAY
FG_GREY
FG_RED
FG_GREEN
FG_BROWN
FG_YELLOW
FG_BLUE
FG_MAGENTA
FG_CYAN
FG_BRIGHT_WHITE

The values are set using the tput statement from the terminfo project, so they are compatible with whatever terminal you’re using. These days most terminals use the ANSI sequences, but if you’re running an old oddball serial terminal it may use different codes.

To use, source it in your script, then use the colour names in echo statements:

1
2
3
4
#!/bin/bash
source /usr/local/bin/set-fg-colour-vars
echo "This text is ${FG_GREEN}green${FG_WHITE}."
echo "This text is ${FG_RED}red${FG_WHITE}."

Code: set-fg-colour-vars


show_env.sh: write current environment to a file

show_env.sh dumps environment variables and other user-specific information to /tmp/show_env.out. I orginally wrote it so I could see the environment for cron jobs. It’s also useful for programs being run from another program such as Apache CGI or a text editor.

Code: show_env.sh


swapfn: switch the names of two files

Given two file names (e.g. foo.txt and bar.txt), renames foo.txt to bar.txt and bar.txt to foo.txt. Works for directories as well.

Code: swapfn


trquo: translate straight quote marks to curly quotes

trquo intelligently translates straight quote marks (both single and double) favoured by plain text editors and most web pages into curly quotes according to conventions common in modern written English. Accepts input from one or more files named on the command line or on stdin; writes output to stdout. Automatically detects HTML files; in HTML mode does not convert quote marks within tags, comments, stylesheets, JavaScript, or <code> blocks, which might break the rendering.

Known bugs:

  • trquo writes only UTF-8 output: it doesn’t handle other common formats such as ISO-8859-1.
  • trquo does not recognize TeX-like conventions where `` is an opening double-quote and '' is a closing double-quote.
  • There are still some edge cases where trquo will render a curly-quote incorrectly. Known cases are quotes surrounding a stretch of whitespace, and quote marks within a stretch of non-alphanumeric characters.

Code: trquo


xbg: run a program in the background

xbg starts a program (typically a GUI program) in the background, writing any output on stdout and stderr to /tmp/xbg-out/program-name.pid.out. I find it useful for starting GUI programs such as a web browser or picture viewer from the command line. At times the output in /tmp/xbg-out is useful for troubleshooting.

Code: xbg


count-http-requests: count HTTP requests for a web page (Firefox)

Reads an input file created from a Firefox Developer Tools Console trace, counts URLs and their success or failure, and displays a list of URLs, counts, and totals. I use this from time to time to see how well ad blockers and NoScript work: I run it once with all the Firefox add-ons disabled, and again by going to the same site with the add-on enabled, and compare the results.

Code: count-http-requests


extract_frames_from_animated_gif

Extracts frames from an animated GIF (as if you couldn’t guess! 😊 ) Writes each extracted frame to its own file in /tmp, then creates simple HTML file that shows each individual frame.

Code: extract_frames_from_animated_gif