Blogs
ed is the standard text editor
ed is a precursor to editors like vi that is focused on editing lines of text. A typical session for someone new to using ed might look something like this:
$ ed help ? quit ? exit ? ? ?
By default ed has no prompt and typing “help” offers you no assistance. Like vi or Vim, it can even be hard to exit. However, ed is a useful tool for editing the output of commands. You can use it as a standalone text editor, but I find little reason to do that. Rather, I use it primarily to edit the output of other commands in a pipeline, especially when I’m working in Emacs Org mode.
Below is a quick example of a real ed session:
$ ed test.txt test.txt: No such file or directory a this is a line this is another time to save and exit . w 53 q
First, you can see ed complain at me that the file doesn’t exist. Like Vim, however, once I’m in the file, I can append to it with the a command. Then, you can see the three lines I typed into the file displayed directly onto the screen. Instead of pressing Escape like vi or Vim, you exit insert mode by typing just a period on a line. Then I write the file with w, which displays the number of characters written, before exiting with q.
As you may notice, ed probably works best when you are creating a new file or when you are very familiar with the contents of an existing file. You can print the contents of a file using the ,p command, which takes advantage of the fact that ranges default to start at the first line and end at the last line. In other words that command is equivalent to 1,$p, where 1 is the first line and $ is the way to denote the last line. Again, this type of range should be familiar if you have used vi or Vim.
You can also use the -p flag to set a prompt to make the session a little more clear. Here you can see the effects of setting the prompt along with the p command and the n command, which prefixes the line by its line number.
$ ed -p "ed> " test.txt 53 ed> ,p this is a line this is another time to save and exit ed> n 3 time to save and exit ed>
ed has many other commands that I’m not even familiar with, but most of them should be at least vaguely familiar if you use Vim. Now, what I really want to show off here is the ability to use ed as a part of pipelines. To reach back to my last blog post on using tee and process substitution in Bash, what I used tee for there was to do some very awkward wrangling of my script’s output with head and tail. All I really wanted to do was move three lines from the bottom of the output to the top, and I could not find a good way to do it. I know now that I could have used AWK arrays to store the lines for printing, printed the header, and then looped through the array to print the job lines. But that’s not a good way to learn to use ed!
Here is the command I ended up going with at the end of that post:
qstat | awk 'BEGIN {q = r = e = 0} FNR == 1 {print} \ (/USER/ && / [QRE] /) {if (/Q/) q++; if (/R/) r++; \ if (/E/) e++; print} END \ {printf "%d in queue\n%d running\n%d exiting\n", q, r, e}' \ | tee >( head -n -3 ) >( tail -n 3 ) > /dev/null
tee is obviously not the right tool for the job (but it did give me an opportunity to learn to use it), as I am having to redirect part of its output to /dev/null. Instead, I can easily use ed for this. One of the ed commands I did not mention yet is the m command, which moves a line somewhere else and sounds quite promising for what I want. The output I was getting in the last post from the awk portion looked like
Job id Name User Time Use S Queue 12345 jobName brent 00:00 R Q1 0 in queue 0 running 0 exiting
And I wanted the summary of which jobs were in queue, running, and exiting at the top for the cases where I have hundreds of jobs running and just want to pipe the whole thing into head to see the oldest ones. As an aside, ed also allows you to match lines by regular expressions rather that by numbers. I believe this is the origin of the name of the grep command since the syntax for doing this kind of search and printing the result in ed is g/re/p, where re is a regular expression. Like in Vim, the g denotes a global search, the slashes delimit the regular expression, and p is the print command we already saw.
Anyway, what I wanted to do was grab the last three lines and move them to the top, so the regular expression matching was actually irrelevant for me then but will come in handy in another example. The ed command I need to run is $-2,$m0, where $ again is the last line, the -2 is a negative offset from $ to select the third to last line, the comma lets me build a range, m is the move command, and 0 is where I want to put it. You can’t jump to line 0 in ed, but if you want to put something before the real first line, you are allowed to move it to the zeroth.
Now, the problem I had was how to get it into ed. The example I’ve shown you so far was straightforward enough because you just open ed like any other text editor and go to work. This may not sound like a problem because you probably know we can pipe the output of a command into a text editor. However, the issue is that we also need to pipe in a list of commands for ed to run, in this case $-2,$m0. Since I was in a process substitution mood when first working on this, the command I came up with involved putting the whole qstat and awk pipeline into a process substitution as the input to the ed command like
and piping the move command into ed from the other side. This turned out to be a decent approach, but I had some trouble with it as well. My first attempt looked like
If you are a bash expert, you will pretty rightly be making fun of me right now, but I just could not figure out what the problem was! I actually made a StackExchange post asking for help because I got so frustrated. Fortunately, the users there set me straight and reminded me that double quotes allow bash to substitute variables, while single quotes do not. This is a problem because variables in bash are referenced with the $, so here I am really giving ed the command himBHs2, because $- contains the options for the current shell, which accounts for the himBHs part, the 2, are not part of the variable expansion, so they are passed verbatim, and then $m0 is an empty variable, so it is printed as nothing.
As I mentioned before, replacing the double quotes with single quotes will fix the problem, and the command
does almost do what I want. There are actually a couple more pieces you might remember from the interactive ed sessions earlier. Namely, we need to actually print the lines after moving them to their new positions, and then we need to exit. One thing to point out is that the ed analog of Vim’s q! or quit and discard changes command is Q, which I will use here since I just want to print the results and then discard them. So the real command using echo looks like
where I have had to use the -e flag to enable the interpretation of escape sequences. Apparently this is not portable or something because the people on StackExchange said it was better to use printf for this. printf is pretty nice too because you can give it one format string and it will apply it to all the other arguments you pass, so using printf the command becomes
That’s the version currently in my script on the computer system I wanted that script for. However, in looking at ed’s fairly limited manual page in attempting to use it again the other day, I noticed that it actually takes the ! option at the command line, allowing it to read the output of a shell command, much like in Vim. With that in mind, I could actually dispense with the process substitution and make the final command
where I have also added the -s flag to suppress the usual diagnostic messages printed by ed. I think the main annoying one is the number of characters read in initially. There seems to be no difference in speed between the two of these, but I find the version without process substitution a bit more ergonomic, and that should make it portable to shells without process substitution built in. With reference to portability, I should emphasize the potential importance of the '! ... ' scheme. On the computer this script was meant for and within my Org source blocks, I actually used !"...", but that is not working for me within my actual terminal. If you run into trouble, you may need to play with these different quoting schemes. If the command you are running does not involve a pipe, I think you can get away without quoting at all.
Now, for a quick example using ed’s regular expression matches, I have another real use of ed that came up in building some data tables for a paper I was working on. One of our collaborators sent a fairly complex directory, and I was supposed to search through it, grab the important values, and format them into tables for our Supplemental Information. Of course I was working in Emacs and specifically in Org mode since I like how easy it is to modify Org tables and then export them to LaTeX. As a result, I created an Org shell code block and went to work. First I had to find the frequency output files using the find command:
$ find . -iname "freq.out" ./cc-pV5Z/MP2/freq.out ./aug-cc-pVDZ/MP2/freq.out ./cc-pVDZ/MP2/freq.out ./aug-cc-pVTZ/MP2/freq.out ./cc-pCVDZ/MP2_fc/freq.out ./cc-pCVDZ/MP2/freq.out ./aug-cc-pV5Z/MP2/freq.out ./cc-pCVQZ/MP2_fc/freq.out ./cc-pCVQZ/MP2/freq.out ./cc-pCV5Z/MP2_fc/freq.out ./cc-pCV5Z/MP2/freq.out ./cc-pV6Z/MP2/freq.out ./cc-pCVTZ/MP2_fc/freq.out ./cc-pCVTZ/MP2/freq.out ./cc-pVTZ/MP2/freq.out ./aug-cc-pVQZ/MP2/freq.out ./cc-pVQZ/MP2/freq.out
As you can see, the results are pretty messy since I wanted to gather the frequencies in groups of similar results. For those unfamiliar with quantum chemistry, I think it should still be pretty clear which go together. For example, I wanted all of the cc-pV.Z directories grouped together, all of the aug-cc-pV.Z ones together, and so on. Piping into sort gets me part of the way there as you can see
./aug-cc-pV5Z/MP2/freq.out ./aug-cc-pVDZ/MP2/freq.out ./aug-cc-pVQZ/MP2/freq.out ./aug-cc-pVTZ/MP2/freq.out ./cc-pCV5Z/MP2_fc/freq.out ./cc-pCV5Z/MP2/freq.out ./cc-pCVDZ/MP2_fc/freq.out ./cc-pCVDZ/MP2/freq.out ./cc-pCVQZ/MP2_fc/freq.out ./cc-pCVQZ/MP2/freq.out ./cc-pCVTZ/MP2_fc/freq.out ./cc-pCVTZ/MP2/freq.out ./cc-pV5Z/MP2/freq.out ./cc-pV6Z/MP2/freq.out ./cc-pVDZ/MP2/freq.out ./cc-pVQZ/MP2/freq.out ./cc-pVTZ/MP2/freq.out
but I still wanted to separate the _fc directories from the other ones, and I wanted them in ascending basis set size, that is TZ, QZ, 5Z, 6Z. I also wanted to delete the DZ results since those weren’t really worth reporting. With the earlier example in mind, we can start with deleting the DZs in a very similar ed command.
printf "%s\n" 'g/DZ/d' ',p' 'Q' | ed -s '!find . -iname freq.out | sort' ./aug-cc-pV5Z/MP2/freq.out ./aug-cc-pVQZ/MP2/freq.out ./aug-cc-pVTZ/MP2/freq.out ./cc-pCV5Z/MP2_fc/freq.out ./cc-pCV5Z/MP2/freq.out ./cc-pCVQZ/MP2_fc/freq.out ./cc-pCVQZ/MP2/freq.out ./cc-pCVTZ/MP2_fc/freq.out ./cc-pCVTZ/MP2/freq.out ./cc-pV5Z/MP2/freq.out ./cc-pV6Z/MP2/freq.out ./cc-pVQZ/MP2/freq.out ./cc-pVTZ/MP2/freq.out
Since the aug results are in reverse order, I can add a g/aug/m0 command to the ed input to move them into the right order at the beginning of the output. I can play a similar trick on the others, along with a direct move on the 6Z to at least get them into chunks. I can do the same thing if I really want careful control over where each chunk is, but this is already pretty handy. The final command and output is
$ printf "%s\n" 'g/DZ/d' 'g/aug/m0' 'g/pCV/m0' 'g/_fc/m$' \ 'g/\/cc-pV.Z\//m0' 'g/6Z/m4' ',p' 'Q' | ed -s '!find . -iname freq.out \ | sort' ./cc-pVTZ/MP2/freq.out ./cc-pVQZ/MP2/freq.out ./cc-pV5Z/MP2/freq.out ./cc-pV6Z/MP2/freq.out ./cc-pCVTZ/MP2/freq.out ./cc-pCVQZ/MP2/freq.out ./cc-pCV5Z/MP2/freq.out ./aug-cc-pVTZ/MP2/freq.out ./aug-cc-pVQZ/MP2/freq.out ./aug-cc-pV5Z/MP2/freq.out ./cc-pCVTZ/MP2_fc/freq.out ./cc-pCVQZ/MP2_fc/freq.out ./cc-pCV5Z/MP2_fc/freq.out
Hopefully this taught you something about ed! I’ve basically only heard about ed in the “standard text editor” memes, and those are pretty fair since the documentation is pretty lacking in the manual page. The info page is a lot more substantial, so if this piques your interest check it out. Obviously it’s not going to replace Vim or Emacs as a main text editor, but it is a nice tool to have in your belt for scripting and especially for formatting Org source block output for use in other blocks.