FRET is a *nix command line tool that examines any file or files and attempts to identify the data structures and patterns within those file(s). It does this by firstly scanning files using heuristic algorithms to identify structures and then by comparing files to identify common patterns. Results are compared and ranked using a probabilistic approach. It outputs a sorted list of the detected structures within the file(s) which can then be parsed by other tools and scripts. libfret is the library used to implement FRET. It provides a clear interface to allow for the analysis of buffers of raw data. It is designed to be integrated with a range of other programs such as hex editors, packet analysers and other development tools.
FRET and libfret are at an early development stage and do not yet provide a lot of useful functionality. FRET (and libfret) scans a single or multiple files (buffers) and currently identifies the following structures:
- data that is common to multiple files or buffers
- arrays of ASCII bytes
- arrays of fill bytes
- offsets to other fields in a file
Development is currently underway for this functionality;
- improve "grinding" scan results
- improve amalgamation of detected structures for a group of files
- verification of the statistical model
- add new scans for other types text encodings
- add new scans for CPU instructions
- add multi-threading support to libfret
- include muti-language support
Provide an additional tool to developers which is capable of identifying major structures within all types of files, without any knowledge of specific file formats.
FRET and libfret are currently developed for the GNU/Linux platform using gcc and GNU autotools. libfret and FRET currently have no dependencies (libredblack and the FNV hash code are included as source). It is not planned to add support for non-POSIX platforms.
FRET and libfret are both released under the GPL open source license. See the project file COPYING for more information on the license.
Thanks to Damian Ivereigh for his excellent Red-Black Tree implementation called libredblack. Multiple Red-Black Trees are used for internal Gram storage. More information is available at the libredblack website.
FNV Hash Algorithm
Thanks to Landon Curt Noll for making the FNV (Fowler-Noll-Vo) hash source code available in the Public Domain. More information is available at Landon's website.