aboutsummaryrefslogtreecommitdiff
path: root/src/split_for_sort.c
Commit message (Collapse)AuthorAgeFilesLines
* split_for_sort: Append mode implementedThorsten Töpper2025-08-291-16/+54
| | | | | | | There may be situations when not every input file is available at once, so those can't be handled in a single session. The append mode opens the files without overwriting the previous content so making the way the tool can be used in scripts more flexible.
* split_for_sort: set RLIMIT_NOFILE to maxThorsten Töpper2025-08-231-0/+25
|
* split_for_sort: make use of FS cacheThorsten Töpper2025-08-231-3/+0
|
* split_for_sort: add filename to warningThorsten Töpper2025-08-111-1/+1
|
* split_for_sort: switch from strncpy to memcpyThorsten Töpper2025-08-101-4/+4
|
* split_for_sort: Split a given file into bucketsThorsten Töpper2025-08-101-0/+367
The target bucket is decided based on the first X characters of a line. The bucket name gets a prefix defined as argument and can be sorted faster on weak hardware. Note: This is just a split alternative. Real world usage in a shell script with a file in which the first 10 characters are the equal in each line, the following 2 bytes are evaluated for splitting: split_for_sort TMPSFS 12 raw_data.txt for f in TMPSFS ; do sort -o "${f}_sorted" -u "${f}" done \# Rely on the argument resolution to go with lexical order cat TMPSFS*_sorted > sorted_data.txt rm TMPSFS*