[linux] sort and unquie by multiple column using awk 根據多欄位去除重複行

[linux] sort and unquie by multiple column using awk 根據多欄位去除重複行

sort 根據多個欄位排序

sort -t 指定分隔欄位 -k 可以指定欄位 -r 逆序
  -t, --field-separator=SEP
              use SEP instead of non-blank to blank transition
  -k, --key=KEYDEF
              sort via a key; KEYDEF gives location and type
  -r, --reverse
              reverse the result of comparisons


 cat a | sort -k1,1
A 1 1
B 1 1
B 2 1

 cat a | sort -k1,1
A 1 1
B 1 1
B 2 1

 cat a | sort -k1,1 -k2,2
A 1 1
B 1 1
B 2 1

 cat a | sort -k1,1 -k2,2nr
A 1 1
B 2 1
B 1 1

 cat a | sort -k1,1 -k2,2nr | awk '!_[$1]++'
A 1 1
B 2 1

在配合使用 awk 過濾已經排序後的行

awk '!_[$1]++'
_ 一個新的hash
$1根據第1欄


cat b | sort -k1,1 -k2,2nr               
A 2 1
A 1 1
B 2 1
B 1 1

awk -F"," '!_[$1]++' file
-F sets the field separator.
$1 is the first field.
_[val] looks up val in the hash _(a regular variable).
++ increment, and return old value.
! returns logical not.
there is an implicit print at the end.