The Text Processing Cookbook

Tools and techniques for processing text and data on the command line

by Jud Dagnall

Contents

• An Introduction
  • Other Resources
• Filter and Select
  • ag - the silver searcher
  • ripgrep - a modern search tool
  • searching via perl
• Extraction
  • Extracting one or more columns with awk
  • Field extraction via perl -anE
  • Extract simple fields via cut
  • Extract by character position with cut
  • Extract fixed-width fields with awk
  • Extract fixed-width fields with in2csv
  • Convert whitespace-delimited columns to csv
  • cutting columns, other tools
  • f - trivial field extractor
  • scut - swiss army knife of column cutters
  • to extract columns from CSV data, use csvcut
• Transformation
  • General transformation with perl -pE and -nE
  • Create several simple filters rather than one complicated one
  • collapse or replace spaces and newlines
  • convert spaces to newline with tr or perl
  • remove newlines with perl
  • reshape text with rs
  • merge sort multiple files of sorted data
  • paste: add files side by side
  • join: intersect two files
  • Concatenate files, skipping header line
  • Remove the first n lines of a file with tail
  • Sort a file with a header
  • put data into a specific number of columns with pr
  • making data tables with column
  • Use column to create a flexible number of columns to fill the width.
  • joining all lines with xargs or paste
  • joining/transforming all except the last line with perl
  • Transform one column at a time
• Grouping Data
  • Find distinct items, removing duplicates
  • Find distinct items without sorting
  • Find duplicate items
  • Find duplicate items without sorting
  • Find lines that are in one file, but not in another
  • Split data into files based on a field
• Frequency Counts and Distributions
  • get a frequency count of items, or find common items
  • Find the n most common items
  • Better frequency counts
  • Histogram of values
• Specialized Tools for Aggregation, Summary, Analysis and Reporting
  • stats
  • csvstat
  • datamash
• CSV and TSV
  • csvkit
• json
  • jq
• Sorting
  • gnusort on osx via coreutils
• Generating Data
  • Generating columns of data by column
  • Generating columns of data by row
  • Generating a sequence of letters:
  • Generating random numbers
  • Generating permutations with shuf
  • Generating date sequences
• Batch and parallel execution with xargs and parallel
  • xargs
  • GNU parallel
• Visualization
  • Sparklines
• Misc
  • Progress bars in pipes
• Solutions: Putting it all together
  • Find available data by user, event class and date