ChatGPT解决这个技术问题 Extra ChatGPT

How can I output only captured groups with sed?

Is there a way to tell sed to output only captured groups?

For example, given the input:

This is a sample 123 text and some 987 numbers

And pattern:

/([\d]+)/

Could I get only 123 and 987 output in the way formatted by back references?

Note, group capture requires sed to turn on extended regular expressions with the -E flag.
Also note, sed -E is for Max OSX and FreeBSD. If you are using a GNU distro (or in Git Bash or WSL), sed -r also works. If you're concerned about cross-platform compatibility, prefer -E.

D
Dennis Williamson

The key to getting this to work is to tell sed to exclude what you don't want to be output as well as specifying what you do want. This technique depends on knowing how many matches you're looking for. The grep command below works for an unspecified number of matches.

string='This is a sample 123 text and some 987 numbers'
echo "$string" | sed -rn 's/[^[:digit:]]*([[:digit:]]+)[^[:digit:]]+([[:digit:]]+)[^[:digit:]]*/\1 \2/p'

This says:

don't default to printing each line (-n)

exclude zero or more non-digits

include one or more digits

exclude one or more non-digits

include one or more digits

exclude zero or more non-digits

print the substitution (p) (on one line)

In general, in sed you capture groups using parentheses and output what you capture using a back reference:

echo "foobarbaz" | sed 's/^foo\(.*\)baz$/\1/'

will output "bar". If you use -r (-E for OS X) for extended regex, you don't need to escape the parentheses:

echo "foobarbaz" | sed -r 's/^foo(.*)baz$/\1/'

There can be up to 9 capture groups and their back references. The back references are numbered in the order the groups appear, but they can be used in any order and can be repeated:

echo "foobarbaz" | sed -r 's/^foo(.*)b(.)z$/\2 \1 \2/'

outputs "a bar a".

If you have GNU grep:

echo "$string" | grep -Po '\d+'

It may also work in BSD, including OS X:

echo "$string" | grep -Eo '\d+'

These commands will match any number of digit sequences. The output will be on multiple lines.

or variations such as:

echo "$string" | grep -Po '(?<=\D )(\d+)'

The -P option enables Perl Compatible Regular Expressions. See man 3 pcrepattern or man 3 pcresyntax.


As a note, OSX Mountain Lion no longer supports PCRE in grep.
Ask your sysadmin to install gsed. You'd be amazed at what a few donuts will get you...
Note that you might need to prefix the '(' and ')' with '\', I don't know why.
@lumbric: If you're referring to the sed example, if you use the -r option (or -E for OS X, IIRC) you don't need to escape the parentheses. The difference is that between basic regular expressions and extended regular expressions (-r).
I found the accepted answer confusing b/c it incorporated a large regexp with the example, making it hard to extract the needed information: In sed you must escape parenthesis \(.*\), access capture groups with \1, \2, ect..
P
Peter McG

Sed has up to nine remembered patterns but you need to use escaped parentheses to remember portions of the regular expression.

See here for examples and more detail


sed -e 's/version=\(.+\)/\1/' input.txt this will still output the whole input.txt
@Pablo, In your pattern you have to write \+ instead of +. And I dont understand why people use -e for just one sed command.
use sed -e -n 's/version=\(.+\)/\1/p' input.txt see: mikeplate.com/2012/05/09/…
I'd suggest using sed -E to use the so-called "modern" or "extended" regular expressions that look a lot closer to Perl/Java/JavaScript/Go/whatever flavors. (Compare to grep -E or egrep.) The default syntax has those strange escaping rules and is considered "obsolete". For more info on the differences between the two, run man 7 re_format.
g
ghostdog74

you can use grep

grep -Eow "[0-9]+" file

@ghostdog74: Absolutely agree with you. How can I get greo to output only captured groups?
@Michael - that's why the o option is there - unixhelp.ed.ac.uk/CGI/man-cgi?grep : -o, --only-matching Show only the part of a matching line that matches PATTERN
@Bert F: I understand the matching part, but it's not capturing group. What I want is to have like this ([0-9]+).+([abc]{2,3}) so there are 2 capturing groups. I want to output ONLY capturing groups by backreferences or somehow else.
Hello Michael. Did you managed to extract nth captured group by grep ?
@Pablo: grep's only outputting what matches. To give it multiple groups, use multiple expressions: grep -Eow -e "[0-9]+" -e "[abc]{2,3}" I don't know how you could require those two expressions to be on one line aside from piping from a previous grep (which could still not work if either pattern matches more than once on a line).
佚名

run(s) of digits

This answer works with any count of digit groups. Example:

$ echo 'Num123that456are7899900contained0018166intext' \
   | sed -En 's/[^0-9]*([0-9]{1,})[^0-9]*/\1 /gp'

123 456 7899900 0018166

Expanded answer.

Is there any way to tell sed to output only captured groups?

Yes. replace all text by the capture group:

$ echo 'Number 123 inside text' \
   | sed 's/[^0-9]*\([0-9]\{1,\}\)[^0-9]*/\1/'

123
s/[^0-9]*                           # several non-digits
         \([0-9]\{1,\}\)            # followed by one or more digits
                        [^0-9]*     # and followed by more non-digits.
                               /\1/ # gets replaced only by the digits.

Or with extended syntax (less backquotes and allow the use of +):

$ echo 'Number 123 in text' \
   | sed -E 's/[^0-9]*([0-9]+)[^0-9]*/\1/'

123

To avoid printing the original text when there is no number, use:

$ echo 'Number xxx in text' \
   | sed -En 's/[^0-9]*([0-9]+)[^0-9]*/\1/p'

(-n) Do not print the input by default.

(/p) print only if a replacement was done.

And to match several numbers (and also print them):

$ echo 'N 123 in 456 text' \
  | sed -En 's/[^0-9]*([0-9]+)[^0-9]*/\1 /gp'

123 456

That works for any count of digit runs:

$ str='Test Num(s) 123 456 7899900 contained as0018166df in text'
$ echo "$str" \
   | sed -En 's/[^0-9]*([0-9]{1,})[^0-9]*/\1 /gp'

123 456 7899900 0018166

Which is very similar to the grep command:

$ str='Test Num(s) 123 456 7899900 contained as0018166df in text'
$ echo "$str" | grep -Po '\d+'
123
456
7899900
0018166

About \d

and pattern: /([\d]+)/

Sed does not recognize the '\d' (shortcut) syntax. The ascii equivalent used above [0-9] is not exactly equivalent. The only alternative solution is to use a character class: '[[:digit:]]`.

The selected answer use such "character classes" to build a solution:

$ str='This is a sample 123 text and some 987 numbers'
$ echo "$str" | sed -rn 's/[^[:digit:]]*([[:digit:]]+)[^[:digit:]]+([[:digit:]]+)[^[:digit:]]*/\1 \2/p'

That solution only works for (exactly) two runs of digits.

Of course, as the answer is being executed inside the shell, we can define a couple of variables to make such answer shorter:

$ str='This is a sample 123 text and some 987 numbers'
$ d=[[:digit:]]     D=[^[:digit:]]
$ echo "$str" | sed -rn "s/$D*($d+)$D+($d+)$D*/\1 \2/p"

But, as has been already explained, using a s/…/…/gp command is better:

$ str='This is 75577 a sam33ple 123 text and some 987 numbers'
$ d=[[:digit:]]     D=[^[:digit:]]
$ echo "$str" | sed -rn "s/$D*($d+)$D*/\1 /gp"
75577 33 123 987

That will cover both repeated runs of digits and writing a short(er) command.


Surprised after reading the high voted accepted answer, I scrolled down to write about its narrow scope and to actually address the spirit of the question. I should have guessed that someone would have done it years ago already. This is very well explained and is the true correct answer.
This is a little hacky and doesn't generalise well. The problem with this approach is that the pattern [^0-9]*([0-9]+)[^0-9]* needs to be designed in such a way that it never crosses the boundary of another match. That works OK for this example, but for complex search queries that don't work on a character-by-character basis, it isn't very practical to have to surround the actual desired match group (whatever) which its forward-lookup and reverse-lookup negation.
It also needs to capture everything that is not part of the capture groups.
C
Ciro Santilli Путлер Капут 六四事

Give up and use Perl

Since sed does not cut it, let's just throw the towel and use Perl, at least it is LSB while grep GNU extensions are not :-)

Print the entire matching part, no matching groups or lookbehind needed: cat <

Single match per line, often structured data fields: cat <

Multiple fields: cat <

Multiple matches per line, often unstructured data: cat <


What didn't you get with the end of the question : "with sed" ?
@Moonchild Googlers don't care.
i found this useful. not all command line regex problems need to be solved with sed.
J
Joseph Quinsey

I believe the pattern given in the question was by way of example only, and the goal was to match any pattern.

If you have a sed with the GNU extension allowing insertion of a newline in the pattern space, one suggestion is:

> set string = "This is a sample 123 text and some 987 numbers"
>
> set pattern = "[0-9][0-9]*"
> echo $string | sed "s/$pattern/\n&\n/g" | sed -n "/$pattern/p"
123
987
> set pattern = "[a-z][a-z]*"
> echo $string | sed "s/$pattern/\n&\n/g" | sed -n "/$pattern/p"
his
is
a
sample
text
and
some
numbers

These examples are with tcsh (yes, I know its the wrong shell) with CYGWIN. (Edit: For bash, remove set, and the spaces around =.)


@Joseph: thanks, however, based on my task I feel like grep is more natural, like ghostdog74 suggested. Just need to figure out how to make grep output the capture groups only, not the whole match.
Just a note, but the plus sign '+' means 'one or more' which would remove the need for repeating yourself in the patterns. So, "[0-9][0-9]*" would become "[0-9]+"
@RandomInsano: In order to use the +, you would need to escape it or use the -r option (-E for OS X). You can also use \{1,\} (or -r or -E without the escaping).
B
Bert F

Try

sed -n -e "/[0-9]/s/^[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\).*$/\1 \2 \3 \4 \5 \6 \7 \8 \9/p"

I got this under cygwin:

$ (echo "asdf"; \
   echo "1234"; \
   echo "asdf1234adsf1234asdf"; \
   echo "1m2m3m4m5m6m7m8m9m0m1m2m3m4m5m6m7m8m9") | \
  sed -n -e "/[0-9]/s/^[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\).*$/\1 \2 \3 \4 \5 \6 \7 \8 \9/p"

1234
1234 1234
1 2 3 4 5 6 7 8 9
$

e
eskogh

You need include whole line to print group, which you're doing at the second command but you don't need to group the first wildcard. This will work as well:

echo "/home/me/myfile-99" | sed -r 's/.*myfile-(.*)$/\1/'

T
Thomas Bratt

It's not what the OP asked for (capturing groups) but you can extract the numbers using:

S='This is a sample 123 text and some 987 numbers'
echo "$S" | sed 's/ /\n/g' | sed -r '/([0-9]+)/ !d'

Gives the following:

123
987

S
Sida Zhou

I want to give a simpler example on "output only captured groups with sed"

I have /home/me/myfile-99 and wish to output the serial number of the file: 99

My first try, which didn't work was:

echo "/home/me/myfile-99" | sed -r 's/myfile-(.*)$/\1/'
# output: /home/me/99

To make this work, we need to capture the unwanted portion in capture group as well:

echo "/home/me/myfile-99" | sed -r 's/^(.*)myfile-(.*)$/\2/'
# output: 99

*) Note that sed doesn't have \d


P
Patrick Häcker

You can use ripgrep, which also seems to be a sed replacement for simple substitutions, like this

rg '(\d+)' -or '$1'

where ripgrep uses -o or --only matching and -r or --replace to output only the first capture group with $1 (quoted to be avoid intepretation as a variable by the shell) two times due to two matches.