Decomp integration

This is an RFC (request for comments). It is not currently an available feature in SkyTemple, but rather a place to gather thoughts and ideas for a future feature.

Support for reading and writing from external files (in addition to the ROM binary) has been a recurring topic of discussion. Now that the pmd-sky decomp is shiftable and technically can be used to make ROM hacks, SkyTemple can leverage this to bootstrap external file support while also gaining the hacking conveniences of working with a decomp. The old way of directly reading/writing from a ROM binary should remain for backwards compatibility and users who prefer the existing workflow.

Motivation

External file support

• Human-readable files are easier to view/edit with the user's favorite editor or scripting language without having to write a custom tool in SkyTemple's UI or CLI.
• Separate files allow the use of Git and other source control tools, with the usual collaboration and versioning benefits that come with version control.

Using the decomp

• C injection without needing a separate tool (C of Time).
• Adding/removing lines of ASM without breaking the ROM, since the decomp will shift all pointers automatically. This removes the need for ASM hacks like overlay 36.
• As the decomp progresses, functions will start to be editable in C without needing to know ASM.
• As the decomp progresses, functions can be documented at a more granular level with variable names and code comments, in addition to the symbol-level documentation offered by pmdsky-debug today.

Implementation

SkyTemple currently requires an EoS ROM to read data and write changes. In addition to allowing a ROM, SkyTemple can allow the user to select a folder containing a pmd-sky decomp repository, which it can read data from and write data to instead of interacting with the ROM directly. The decomp contains all necessary information to build a matching EoS ROM, so the user will not need to supply a separate ROM if they supply the decomp.

SkyTemple's UI can remain the same regardless of data source. A shared interface for reading and writing data can be used such that there is a set of file handlers using the raw ROM and another set of file handlers using the decomp. This will abstract out the IO to the data source and ensure that operations in SkyTemple's UI don't need to care which data source is used.

NitroFS file system

In the decomp, the full NitroFS file system is located in the files folder in the project. Right now, these files are in their raw binary forms. SkyTemple can read/write directly to these binary files for now, but ideally these files could be extracted out to more human-readable formats where appropriate (e.g., the PMD Red decomp has a monster_data.json. This human-readable format can be embedded directly in the decomp project, and the build scripts can be updated to transform these human-readable files into their binary counterparts before building the ROM. If SkyTemple needs to add additional files to the file system, it should also add the file names to nitrofs_files.txt.

For most key/value byte data, a standard serialization language like YAML or JSON will be sufficient.

Script files

ExplorerScript does not compile 1-1 to the script files in the original game, so it cannot be used by the decomp. However, the sub-language SsbScript compiles 1-1, and can be embedded in the decomp if desired. It may also be possible to add optional functionality to dump ExplorerScript files when creating a hack, as the user no longer needs the 1-1 match at this point.

Assembly code

There are pieces of data embedded in the ASM that SkyTemple will need to know the locations of. When the user starts changing the ASM, this can shift pointers from their original locations, so SkyTemple can no longer rely on hard-coded pointers when reading data. The xMAP files created by the decomp build (example) contain all symbol addresses, so SkyTemple can use these to locate any data it needs when reading the project. When writing data back, SkyTemple can write to the respective `.s` file containing the data.

Patches

In an ideal world, ASM patches would be obsolete in favor of editing the code directly in the decomp. However, realistically there will need to be compatibility with existing ASM patches. The challenge with ASM patches is hard-coded pointer addresses, which will break if any code change in the decomp shifts the pointers around. There are a couple possible ways to approach this.

• Apply patches on the ROM built by the decomp. This will require a conversion script that looks at the pointer addresses in the patch and shifts them according to the decomp. xMAP files can be used to find the nearest function to a given address in the patch. From here, an offset can be calculated between the patch address and the nearest function address, then added to the patch address to shift it.
• Modify the decomp files based on the patch. Since the decomp contains the raw assembly, a patch file could be incorporated into the built ROM by modifying the decomp's assembly code according to the instructions of the patch file. However, this may run into issues with functions decompiled to C, as they will no longer contain the original assembly code.

Building the project

Unlike a raw ROM, the decomp needs to build the ROM before the ROM can be played. The build is orchestrated via Makefiles, with the `make` command building the ROM based on the current files in the project. It may or may not be useful to have SkyTemple run `make` without the user manually typing the command into a command line, depending on the intended user workflow.

Since the build stage may take time, this has implications for hot-swapping and the debugging process when using SkyTemple's embedded DeSmuME emulator. If any ASM/C code is changed, or on initial setup, the full build process must be run. However, if the only changes are to the NitroFS file system, it might be possible for SkyTemple to hot-swap these changes into the built ROM's files, keeping live emulator support.