_____
 ---'   __\_______
            ______)         

GNU poke

__)

Release notes for poke 2.0

__) ---._______) I am happy to announce a new major release of GNU poke, version 2.0. This release is the result of a year of development. A lot of things have changed and improved with respect to the 1.x series; we have fixed many bugs and added quite a lot of new exciting and useful features. See below for a description of many of them. We have had lots of fun and learned quite a lot in the process; we really wish you will have at least half of that fun using this tool! The tarball poke-2.0.tar.gz is now available at https://ftp.gnu.org/gnu/poke/poke-2.0.tar.gz. GNU poke (http://www.jemarch.net/poke) is an interactive, extensible editor for binary data. Not limited to editing basic entities such as bits and bytes, it provides a full-fledged procedural, interactive programming language designed to describe data structures and to operate on them. Thanks to the people who contributed with code and/or documentation to this release. In certain but no significant order they are: Mohammad-Reza Nabipoor Luca Saiu Bruno Haible Egeyar Bagcioglu David Faust Guillermo E. Martinez Konstantinos Chasialis Matt Ihlenfield Thomas WeiƟschuh Sergei Trofimovich Fangrui Song Indu Bhagat Jordan Yelloz Morten Linderud Sergio Durigan Junior As always, thank you all! User interface updates ~~~~~~~~~~~~~~~~~~~~~~ - Types can now be redefined at the prompt. This allowed us to change `load' to always load the requested file/module. - The poke command now accepts two additional command-line options in order to select which default style to use: --style-dark and --style-bright. The first, which is the default, works good with dark backgrounds. The second works good with bright backgrounds. - Invoking the .set command with no arguments now lists all the global settings along with their values. It also emits buttons (terminal hyperlinks) to easily toggle boolean settings. - The .file dot-command now supports a /c flag, that tells poke it must create an empty file with the specified name if it doesn't already exist. - The :size argument of the dump command is now rounded up to the next byte, not truncated down to the previous byte. - A new "sub" IO space has been implemented in poke. It allows to create IO spaces that are like narrowed versions of some other IO space. These "sub" IO spaces are created using the new dot-command .sub: ,---- | (poke) .mem scratch | (poke) .info ios | Id Type Mode Bias Size Name | * #0 MEMORY rw 0x00000000#B 0x00001000#B *scratch* | (poke) .sub #0, 2, 16, lala | (poke) .info ios | Id Type Mode Bias Size Name | * #1 SUB rw 0x00000002#B 0x00000010#B sub://0/0x0/0x10/lala | #0 MEMORY rw 0x00000000#B 0x00001000#B *scratch* `---- At this point accessing the IO space #1 at offset 0#B will modify the data in the *scratch* IO space at offset 2#B. - GNU poke can now poke at the memory of a running process using the new process IOS. This can be done using the new dot-command .proc: ,---- | (poke) .proc 30244 | (poke) .info ios | Id Type Mode Bias Size Name | * #0 PROC rw 0x00000000#B 0xffffffffffffffff#B pid://30244 `---- The command above has opened a new IO space (with id #0) to poke at the memory of the running process with PID 30244. A flag /m can be passed to .proc to indicate we want poke to create additional sub-spaces providing access to the mapped VM ranges of the process: ,---- | (poke) .proc/m 30244 | (poke) .info ios | Id Type Mode Bias Size Name | #9 SUB r 0x00000000#B 0x00001000#B sub://0/0xffffffffff600000/0x1000/[vsyscall] | #8 SUB r 0x00000000#B 0x00002000#B sub://0/0x7ffe82db2000/0x2000/[vdso] | #7 SUB r 0x00000000#B 0x00002000#B sub://0/0x7ffe82db0000/0x2000/[vvar] | #6 SUB rw 0x00000000#B 0x00021000#B sub://0/0x7ffe82c2f000/0x21000/[stack] | [...] | * #0 PROC rw 0x00000000#B 0xffffffffffffffff#B pid://30244 `---- See below for the new handler syntax in open to open process IO spaces from Poke programs. Note that the support for process IO space is only implemented in GNU/Linux systems. - The dump command now shows ?? marks to denote bytes that are not readable, for whatever reason. This happens for example when we try to access some non-mapped area of the VM space of a process: ,---- | (poke) .proc/m 30244 | (poke) dump :from 0#B :size 16#B | 76543210 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789ABCDEF | 00000000: ???? ???? ???? ???? ???? ???? ???? ???? ................ `---- - The dump command now emits hyperlinks for the shown bytes, if requested. Clicking on them will insert the offset where the byte resides in the poke prompt. - The dump command now remembers the last used offset per IO space. In poke 1.x it just remembered the last globally used offset, which was very confusing. - The .info ios dot-command now emits [close] buttons if hyperlinks are enabled. Clicking on them closes the referred IO space. - The .info ios dot-command now shows the value of the "bias" for each open IO space. See below for more information on this bias. - E_constraint exceptions raised when mapping now include some useful location information: ,---- | (poke) Elf64_Ehdr @ 1#B | unhandled constraint violation exception | constraint expression failed for field Elf_Ident.ei_mag `---- - The .info types, .info variables and .info functions dot-commands now accept an optional regular expression. Example: ,---- | (poke) load elf | (poke) .info types Elf64_ | Name Declared at | Elf64_Ehdr elf-64.pk:184 | Elf64_Off elf-64.pk:26 | Elf64_SectionFlags elf-64.pk:130 | Elf64_Shdr elf-64.pk:152 | Elf64_RelInfo elf-64.pk:36 | [...] `---- When listing types, the names in the first column are emitted with a nice terminal hyperlink. Clicking on them will result in executing .info type NAME. - New dot-command .info type NAME, that prints out a nice and informative description of the type with the given name. Example: ,---- | (poke) .info type Elf64_File | Class: struct | Name: "Elf64_File" | Complete: no | Fields: | ehdr | shdr | phdr | Methods: | get_section_name | get_symbol_name | get_sections_by_name | get_sections_by_type | section_name_p | get_string | get_group_signature | get_group_signatures | get_section_group `---- Poke Language updates ~~~~~~~~~~~~~~~~~~~~~ - Integral structs are struct values that are stored like integers. This is a simple real-life example: ,---- | type Elf_Sym_Info = | struct uint<8> | { | uint<4> st_bind; | uint<4> st_type; | }; `---- These structs can be used pretty much like regular structs, accessing their fields normally. In poke 1.x it was already possible to "integrate" them, i.e. operating with their "integer" value by using either an explicit cast: ,---- | (poke) Elf_Sym_Info { st_bind = 1 } as uint<8> | 0x10UB `---- or automatically, using them in a context where an integral value is expected: ,---- | (poke) Elf_Sym_Info { st_bind = 1} + 1 | 0x11UB `---- We are now introducing support for the inverse operation: to "deintegrate" an integer into an integral struct value. This is performed by a cast: ,---- | (poke) 0x10 as Elf_Sym_Info | Elf_Sym_Info { st_bind=0x1UB, st_type=0x0UB } `---- - Arrays whose elements are integral (integers, other integral arrays, or integral structs) can now be also "integrated" and "deintegrated" using casts: ,---- | (poke) [1UB, 0UB] as int | 0x10 | (poke) 0x10 as uint<8>[2] | [1UB,0UB] | (poke) [[1UB,2UB],[3UB,4UB]] as int | 0x1234 | (poke) 0x1234 as uint<8>[2][2] | [[1UB,2UB],[3UB,4UB]] `---- This nice feature has been contributed by Mohammad Reza-Nabipoor. - Poke programs can now sleep for a number of seconds and nanoseconds using the new built-in function sleep. It has the following signature: ,---- | fun sleep = (int<64> sec, int<64> nsec = 0) void: `---- The sleep is not active, i.e. poke will not consume CPU while it is sleeping. - The only way in poke 1.x to print styled text to the terminal was to use the very ugly and clumsy %<..> tags in printf statements. This was annoying to use, especially because you were required to end every styling class you open in the same statement. So we added a couple of new built-in functions term_begin_class and term_end_class. They are used like this: ,---- | term_begin_class ("error"); | print "error: the quux is foobared"; | term_end_class ("error"); `---- The functions will raise an exception in case you nest your classes the wrong way. - It is now possible to emit terminal hyperlinks from Poke programs. See "Terminal hyperlinks updates" below for more information on this. - The equality `==' and inequality `!=' operators now work on function values. Two given function values are equal if they are the same function value. i.e. given these definitions: ,---- | fun foo = void: {} | fun bar = void: {} `---- In this example foo is equal to foo, but foo is not equal to bar even if both functions happen to have identical type signature and bodies. - Now it is a compile-time error to cast any values to function types. We plan to support this at some point, since it is needed for things like having any fields in structs, but at the moment closure values are not tagged with their type at run-time and it is better for the user to get compile-time errors. - In this release we have changed both the semantics and syntax of struct type field initializers. This change has been motivated by our own practical usage of poke. In poke 1.x a struct field initializer had the form: ,---- | type Foo = | struct | { | int i = 10; | long l; | }; `---- The initialization of i had two effects: newly constructed Foo struct values would have i initialized to 10, and a constraint that i must equal 10 was also implied when mapping structs of type Foo. This worked well, but we realized it is good to decouple the implicit constraint from the initialization value: sometimes you need one of these, but not both. It was also not possible to add additional constraints. So we changed the semantics of the construction above to denote "initialize to 10, but no implicit constraint", with maybe an additional constraint like in: ,---- | type Foo = | struct | { | int i = 10 : i < 100; | long l; | }; `---- And then we added the new syntax: ,---- | type Foo = | struct | { | int i == 10; | long l; | }; `---- to denote "initialize to 10, plus implicit constraint i == 10." - The Poke language is quite exception-oriented. A good example of this is the way to detect whether a given alternative in an union is the currently selected one: refer to it and see if an E_elem exception gets raised. This in practice leads to code like this: ,---- | try { length.indefinite; return 1; } | catch if E_elem { return 0; } `---- This is so common that we have introduced a new "exception conditional" operator ?!. The above code can now be rewritten as: ,---- | return length.indefinite ?! E_elem; `---- The new operator also has a { ... } ?! EXCEPTION form where the code to execute is a compound statement. - Constraints in struct type fields are very often related to the values of other fields in the same struct. The relationship is often in the form "if field X has value N, then I should have a value M". We have added a new "logical implication" operator => (inspired from the recutils operator with the same name) that implements this logic: ,---- | A => B :=: !A || (A && B) `---- For example: ,---- | uint<1> encoding; | uint<5> tag_number : tag_number == BER_TAG_REAL => encoding == 1; `---- Meaning that if tag_number denotes a "real" then encoding must be 1. If tag_number does not denote a "real" then the value of encoding is irrelevant. - Mohammad-Reza Nabipoor has contributed support for a new built-in function format, that is able to format a string out of a format string and a list of values. For example: ,---- | var s = format ("%s - %s (%i32d)", name, sex, age); `---- Will format a string in s like: ,---- | "Francisco Maganto - male (63)" `---- The format string is identical to the one used by printf, and therefore it also accepts %v tags to format nested complex values like arrays and structs. - Exception structs have been expanded to include two additional fields: location and msg. Both new fields are strings, and are used to convey location information (like the field whose constraint expression failed resulting in an E_constraint) and an explanation of why the exception was raised, respectively. There is no more need to abuse the field name for these purposes. - Much like in C, the Poke printf format strings use % to introduce formatting tags, like %i32d. In poke 1.x it was embarrassingly not possible to denote the character % itself. Now %% can be used in printf format strings in order to denote a single % character. - If a struct type has labels and all of them are constant, it is now considered a complete type. i.e. a type whose size is known at compile-time. This makes it possible to use it in sizeof or as the unit of an offset. - The Poke Virtual Machine settings that configure output are now accessible programmatically from Poke using a set of new built-ins: ,---- | vm_{obase,opprint,oacutoff,odepth,oindent,omaps,omode} | vm_set_{obase,opprint,oacutoff,odepth,oindent,omaps,omode} `---- - We added introspection capabilities: typeof. XXX. Standard Poke Library updates ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - The atoi standard function has been improved in order to parse integers with signs. - The ltos standard function now works for any numeration base from 1 to 16, included. - New standard functions opensub and openproc have been introduced to help with the creation of "sub" and process IO spaces, respectively. See below under "IO subsystem updates" for more information. - A new standard function exit has been added to the standard library, that provides a familiar way to exit a Poke program with an optional exit code: ,---- | fun exit = (int<32> exit_code 0) `---- This is just a wrapper to the more "pokeish" and less conventional way of exiting, which is to raise an exception of type EC_exit: ,---- | raise Exception { code = EC_exit, exit_status = exit_code }; `---- - In poke 1.x we forgot to emit an error at compile-time when the user defined an union type as pinned. This obviously doesn't make any sense (all the alternatives of an union are "pinned") so now a compile-time error is raised. - We have realized that it doesn't make much sense for pinned struct types (whose fields all have an implicit offset of zero bytes) to have field labels. To avoid confusing the user, the compiler now emits a compile-time error if such a type is defined. - Unions can no longer have alternatives with labels. This worked in poke 1.x, but had complicated semantics with little real practical benefit. So now a compile-time error is raised if such a type is defined. IO subsystem updates ~~~~~~~~~~~~~~~~~~~~ - Each open IO space now maintains a "bias", which is a bit-offset. This bias is added to the offset specified in every read or write access to the IO space. This bias is programmable, and can be read and set by Poke programs using two new built-in calls: ,---- | fun iobias = (int<32> ios = get_ios) | fun iosetbias = (offset<uint<64,b> bias = 0#b, int<32> ios = get_ios) `---- Note how the positioning and default values of these functions makes it comfortable to use them at the (poke) prompt. - Poke programs can now poke at the standard input, standard output and standard error output of the running poke process using the new stream IO spaces. The new available handlers (to be used with open) are <stdin>, <stdout> and <stderr>. A good application of these IO spaces is to write filter utilities in Poke. This support has been contributed by Egeyar Bagcioglu. - The IOS_F_* flags used in the open builtin has been rationalized. As a consequence the IOS_F_TRUNCATE flag has been removed. - There is a new built-in ioflags that returns the flags used to open some particular IO space. - open now raises an E_perm exception if the user doesn't have enough credentials when opening an IO space. This can happen for example with insufficient permissions when opening a file in some specified mode. - Opening an IO space using a handler <zero> now provides an IO space of size 2^64 bytes whose contents are all zero bytes, and that ignores any writes to it. Example: ,---- | var zeroes = open ("<zero>"); `---- - New IO device sub for sub-spaces. Sub IO spaces that act like a (maybe) narrower version of some other given IO space can be opened using a sub:// handler. Example: ,---- | var sub = open ("sub://2/0x10/0x1000/somename", IOS_M_RDONLY); `---- This will create a sub-space that provides read-only access to the [0x10#B,0x1010#B] range of bytes of some other IO space with id #2. Since it can be annoying to format the handler string, a convenient utility function called opensub has been added to the standard library: ,---- | var sub = opensub (ios, 0x10#B, 0x1000#B, "somename", IOS_M_RDONLY); `---- - New IO device proc for poking at the memory of live processes. The handler to pass to open in order to poke at the memory of a process with a given PID has the form pid://PID. Example: ,---- | var fd = open ("pid://1234"); `---- Since it can be annoying to format the handler string, a convenient utility function called openproc has been added to the standard library: ,---- | var fd = openproc (1234); `---- Terminal hyperlinks updates ~~~~~~~~~~~~~~~~~~~~~~~~~~~ - The payload used in the terminal hyperlinks URL is now reduced to a token number. This makes it more opaque and compact than the previous version where we would encode full Poke expressions in the URL. - The "hyperserver", poke utility subsystem that provides and handles the support for "terminal hyperlinks", has been partially rewritten. The new implementation is mostly written in Poke. This makes it possible for Poke programs (like pickles) to emit and handle terminal hyperlinks. For example: ,---- | var url = hserver_make_hyperlink ('e', "2 + 2"); | | term_begin_hyperlink (url, ""); | print ("[clickme]"); | term_end_hyperlink; `---- Will print a clickable button "[clickme]" in the terminal, which once clicked on will execute the Poke code "2 + 2". - poke 1.x supported two variants of terminal hyperlinks: "execute" and "insert". Clicking on an "execute" hyperlink triggers the execution of some given textual Poke expression or statement. Clicking on an "insert" hyperlink triggers the inclusion of some given string at the current input position at the prompt. We now introduced a third kind of hyperlinks: the "closure" hyperlinks. Once clicked, some given Poke closure/function gets executed. This is particularly useful when generating hyperlinks from a Poke program. For example: ,---- | fun toggle_setting = void: { ... }; | | term_begin_hyperlink (hserver_make_hyperlink ('c', "", (toggle_setting)), ""); | print "[toggle]"; | term_end_hyperlinks; `---- will print a clickable button "[toggle]" in the terminal, which once clicked on will execute the toggle_setting function. libpoke updates ~~~~~~~~~~~~~~~ - poke now installs a pkg-config file poke.pc to ease using libpoke. - libpoke now provides an interface to register foreign IO spaces. This allows adding pokeish capabilities to third-party programs. As an example, poke integration in GDB has been written (not upstreamed yet) that makes it possible to poke the memory of an inferior process being debugged. This is achieved by registering callbacks in the foreign IO interface. - libpoke now provides an inteface to register "alien tokens". This is also useful when integrating libpoke in third-party applications. For example, in the GDB integration this interface has been used to allow users to refer to the value of GDB symbols in Poke programs using this syntax: ,---- | $main `---- and to the address of a given symbol using this syntax: ,---- | $addr::main `---- - It is now possible to create an incremental compiler without standard types. This is useful in cases where the program integrating with libpoke provides its own concept of types like int or long. This is the case of GDB. - We added more services in libpoke to operate on PK values. Many more still need to be added. - libpoke.h is now C++ compatible. Pickles updates ~~~~~~~~~~~~~~~ - New pickle asn1-ber.pk provides definitions to poke ASN-1 data encoded in BER (Basic Encoding Rules.) - New pickles ustar.pk provides definitions to poke USTAR file systems, standardized by POSIX.1-1988 and POSIX.1-2001. - New pickles ctf-dump.pk provides functions that dump the data of CTF sections in a human-readable form. Contributed by Indu Bhagat on behalf of Oracle Inc. - New pickles jffs2.pk provides definitions to poke JFFS2 file systems. - The elf.pk pickle has been splitted in 32-bit and 64-bit variants, and ELF-32 definitions have been added. - The output messages emitted by argp.pk have been improved. - New tests for the btf.pk pickle. Contributed by David Faust on behalf of Oracle Inc. Utilities updates ~~~~~~~~~~~~~~~~~ - New filter pk-strings.pk. - New filter pk-bin2poke.pk. Development tools updates ~~~~~~~~~~~~~~~~~~~~~~~~~ - RAS (the retarded poke assembler) now allows to split long logical lines into several physical lines by finishing them with the backslash character. This is an example: ,---- | .macro struct_field_extractor @struct_type @field @struct_itype \ | @field_type #ivalw #fieldw `---- - RAS now understands the endianness specifiers IOS_ENDIAN_LSB and IOS_ENDIAN_MSB. Better than hardcoded magic numbers. - RAS now emits an error if it finds a .function in which there is not exactly one prolog instruction and at least one return instruction. This problem has bitten me very often, resulting in very puzzling and subtle bugs, glglgl. Notable bug fixes ~~~~~~~~~~~~~~~~~ - The infamous, annoying, hated and despised "PVM_VAL_CLS_ENV (closure) != NULL" bug has (most probably... crossing fingers, toes and whatnot) been finally fixed in this release. I have tried no less than eight times to fix this bug, but not until very recently I finally understood what was going on; I thought I did and the test case would work, but no I didn't and the bug would soon return with renovated hatred. On my defense, the involved area is the hairiest part of the code generator and every time I look at it I basically have to re-learn how it works. Good news is: if the same problem manifests again, now we know how to fix it. But hopefully it is fixed now and I will stop receiving hate email about this. - --disable-hserver now actually disables the hyperserver. Yes, really. Documentation updates ~~~~~~~~~~~~~~~~~~~~~ - We have added a new online help system to the poke utility, which is accessible using the .help dot-command. There are help entries for all the commands, dot-commands and global settings. This new online help system is written in Poke, making it possible to other Poke programs (like pickles) to add their own help topics to the system. - We have expanded the user manual to cover the new functionality and clarify obscure concepts that weren't explained with enough clarity. Despite of this work, the user manual is not yet complete, unfortunately, but getting there. - The user manual now contains a section that explains how to set up your system and terminal emulators to support poke terminal hyperlinks. - There is a new website called Pokology, https://pokology.net, which is maintained by the poke developers and users. It is a live repository of knowledge relative to GNU poke, including practical articles, multimedia stuff, hints and tricks, a frequently asked question, etc. Editor support updates ~~~~~~~~~~~~~~~~~~~~~~ - The Emacs modes poke-mode, poke-ras-mode and poke-map-mode have been update with more features and bug fixes. - We now distribute a module for Poke syntax highlighting for vim. Contributed by Matthew T. Ihlenfield. Other updates ~~~~~~~~~~~~~ - We have rewritten the way global settings (like endianness, numeration base used for output, etc) are handled in both libpoke and the poke application. The new implementation introduces a "settings registry" which is written itself in Poke. As such, other Poke code (like pickles and the like) have full access to the settings registry, and even add their own settings to the application. - Many improvements in compiler error messages. This includes emitting more meaningful messages, for example "got int<32> expected string" instead of "invalid operands in expression". And also better locations, like pointing at the particular operand whose type is wrong instead of the entire expression. Happy poking! -- Jose E. Marchesi Frankfurt am Main 28 February 2022