Cross-compiling NetHack 3.7 Last edit: December 1, 2019

The NetHack 3.7 build process differs from the build process of previous
versions in some important ways that make it possible to use a cross-compiler
running on one platform (the "host" platform of the build) to produce a binary
NetHack package that can execute on an entirely different platform.

Part A Contents:
    A1. Why cross-compile?
    A2. Building NetHack 3.6 (before)
    A3. Building NetHack 3.7 (going forward)
    A4. How was the build procedure reduced to 5 steps?
    A5. How can I help with the cross-compiling initiative?

Part B Contents:
    B1. Two sets of compiles and procedures
    B2. What needs to be built and executed on the HOST?
    B3. What needs to be built for the TARGET?
    B4. Case sample: msdos

--------------------------------------------------------------------------------
 Part A - Cross-compiling NetHack
--------------------------------------------------------------------------------

  +--------------------------+
  | A1. Why cross-compile?   |
  +--------------------------+

By using cross-compilers on host platforms with fast processors, plenty of RAM
and storage resources, and an available cross-compiler, it may be possible to
keep or resurrect a working version of NetHack on platforms that are now too
constrained to carry out the build process natively on the platform any more.

Some of the constraints in carrying out a native build on the desired target
may include, but not necessarily be limited to, any of the following:

  o Access: Somebody with a working knowledge of the NetHack build process may
    not have the desired target build platform available to them. Conversly,
    somebody with a keen knowledge of the target platform, and access to it,
    may not be all that familiar with the NetHack build process.

  o Resources: Address space limitations, insufficient RAM, low amounts of disk
    storage, slow processor performance, may impede the ability to execute the
    compile process on the target platform.

  o Compilers: Some of the native compilers on historical platforms may only
    support the particular dialect of C that was popular when the platform and
    compiler were in their prime. 

Another useful potential result of cross-compiling, is that it paves the way
for carrying out test and production builds of NetHack for multiple target
platforms through automated steps carried out on the host platform(s).


  +-------------------------------------+
  | A2. Building NetHack 3.6 (before)   |
  +-------------------------------------+

Very generally, the build of NetHack in past versions required the following
steps to be carried out:

    1. Compile and link util/makedefs.
    2. Run makedefs repeatedly with different command line options to produce
       several output files that are required for:
        (a) additional build steps to follow, including some header
            files: pm.h, onames.h, date.h.
        (b) creation of files, containing information required by,
            or about the game during its execution, that are stored in a 
            portable, platform-independent way, that need to be inserted
            into the game package.
        (c) creation of files containing information required by, or about
            the game during its execution, that are stored in an architecture
            and/or platform and/or operating system dependent way, that need
            to be inserted into the game package (the quest text format is 
            one example).
    3. Compile and link the level compiler. This step needs to execute 
       work-alike tools to lex and yacc, or needs to build pre-built lex and
       yacc output (.c, .h files) that are provided in the sys/share part of
       the NetHack source code tree.
    4. Execute the level compiler to read dat/*.des files and create
       a set of binary output files that are architecture and/or operating 
       system dependent on the build platform, for use by the game during
       its execution.
    5. Compile and link the dungeon compiler. Like the level compiler, this
       step needs to execute work-alike tools to lex and yacc, or needs to 
       build pre-built lex and yacc output (.c, .h files) that are provided
       in the sys/share part of the NetHack source code tree.
    6. Execute the dungeon compiler to read dat/dungeon.def and create 
       a set of binary output files that are architecture and/or operating
       system dependent on the build platform, for use by the game during
       its execution.
    7. Compile and link several less critical utilities such as uudecode,
       tile-generation utilities, and so forth, all of which need to execute
       on the build platform during the build process to produce output files
       for use during the game, that are reasonably portable (not architecture
       and/or operating system dependent; only the output of the utilities
       becomes part of the game package, not the executable utilities
       themselves.
    8. Compile and link the game itself.
    9. Package the game and its required files including the output from
       previous steps 2b, 2c, 4, 6, 7 and 8 above.

Steps 1, 2a, 2b, 7, and 9 above are not impediments to cross-compiling NetHack.

Steps 2c, 3, 4, 5, 6 and 8 above are impediments to cross-compiling NetHack.

That's because the files that those steps produce are very much tied to the
platform where the build of NetHack is being carried out. Variations between
platforms (such as 32-bit vs 64-bit, integer sizes, pointer sizes, processor
byte order, data alignment requirements, struct padding and the way bitfields
are stored) impact the portability of those data files between different
platforms and operating systems. If all those things happen to match, the files
might, just might, be usable across platforms, but the chances are against it,
and that certainly cannot be counted on.

  +------------------------------------------+
  | A3. Building NetHack 3.7 (going forward) |
  +------------------------------------------+

Again, very generally, the build of NetHack in 3.7 requires the following
steps to be carried out:

    1. Compile and link util/makedefs.
    2. Run makedefs repeatedly with different command line options to produce
       several output files that are required for:
        (a) additional build steps to follow, including some header
            files: pm.h, onames.h, date.h.
        (b) creation of files, containing information required by,
            or about the game during its execution, that are stored in a 
            portable, platform-independent way, that need to be inserted
            into the game package.
    3. Compile and link several less critical utilities such as uudecode,
       tile-generation utilities, and so forth, all of which need to execute
       on the build platform during the build process to produce output files
       for use during the game, that are reasonably portable (not architecture
       and/or operating system dependent; only the output of the utilities
       becomes part of the game package, not the executable utilities
       themselves.
    4. Compile and link the game components for the TARGET; that includes
       NetHack itself, Lua, and any optional regular-expression or window port
       libraries that you plan to link into the NetHack game executable.
    5. Package the game and its required files including the output from
       previous steps 2b, 3 and 4 above.

Step 4 is now the only impediment to cross-compiling NetHack, and is resolved
by executing step 4 using a cross-compiler that runs on the build (host) 
platform to produce a resulting binary for the target platform, instead of
executing the native compiler.

  +-----------------------------------------------------+
  | A4. How was the build procedure reduced to 5 steps? |
  +-----------------------------------------------------+

The following are among several design changes planned in NetHack 3.7,
and these specific changes are what altered the build process to make
cross-compiling possible:

    o  There is no creation of platform-dependent files, such as the quest
       text files, by makedefs during the build process. Instead, the quest
       text files have been converted to Lua and are inserted into the game
       package for processing by the embedded Lua <fixme: interpreter?> 
       during execution of NetHack.

    o  There is no build-time level compiler involved. Instead, the level
       descriptions have been converted to Lua and are inserted into the game
       package for processing by the embeded Lua <fixme: interpreter?> 
       during execution of NetHack. 

    o  There is no build-time dungeon compiler involved. Instead, the dungeon
       description has been converted to Lua and is inserted into the game
       package for processing by the embeded Lua <fixme: interpreter?> 
       during execution of NetHack. 

    o  Some of the build and option information that was formerly produced
       during build time by makedefs, and contained information about the
       build-platform specifically, is now produced at runtime within the
       game under a cross-compiled build. As such, it now produces information
       applicable to the target NetHack environment, not the build environment.

  +------------------------------------------------------------+
  | A5. How can I help with the cross-compiling initiative?    |
  +------------------------------------------------------------+

    o  If you have a favourite target platform (let's call it XX-Platform for
       example purposes) that you'd like to see NetHack be able to run on, do 
       some research to find out if a cross-compiler exists that:
           - produces output for XX-Platform.
           - executes on a platform that you use and love (Linux, Windows,
             Mac OS X are some examples of platforms that have cross-compilers
             for other targets available)

    o  Then, make the community, devteam, and others aware that you're starting
       a cross-compile of NetHack for XX-Platform. You might need to ask some
       "starting out" questions initially, and as you get deeper into it, you 
       might need to ask some tougher questions.
       
    o  Perhaps consider forking from NetHack on GitHub, and do the
       cross-compiler work there in your fork. Strive to get it to a point where
       it's ready to play-test on XX-Platform, or perhaps even use an emulator
       of XX-Platform if one is available. We live in a time where plenty do.

       Doing your work on a GitHub fork has the following advantages:
           - It will make it really simple to integrate your work back into
             the NetHack source tree if that's one of your goals.
           - It will make it possible and straightforward to merge upstream
             NetHack changes into your work for the XX-Platform cross-compile
             so that it stays current with the game as it evolves.
           - You may get help from others in the form of suggestions, or 
             pull-requests, or offers to join the development. Chances are,
             you aren't the only person out there that would like to 
             establish/resurrect/maintain NetHack on XX-Platform.

       Have fun!

-----------------------------------------------------------------------------
 Part B - Cross-compiling details
-----------------------------------------------------------------------------

Part B Contents:
    B1. Two sets of compiles and procedures
    B2. What needs to be built and executed on the HOST?
    B3. What needs to be built for the TARGET?
    B4. Case sample: msdos

  +-----------------------------------------+
  | B1. Two sets of compiles and procedures |
  +-----------------------------------------+

The HOST is the platform/place that you're running the build procedures using
the native compiler/linker, and the cross-compiler/linker that runs on the HOST
to build the game for a TARGET platform.

You have to:
    1. Build mandatory utilities on the HOST.
    2. Execute the mandatory utilities to generate components that will be used
       during step 4 to build the game itself, or that will generate components
       that will become part of the TARGET game package.
    3. Build optional or enhancing utilities on the HOST, execute those
       optional or enhancing utilities on the HOST to generate components that
       will become part of the TARGET game package.
    4. Execute a supported cross-compiler to compile the rest of the game
       components like NetHack, Lua, and any optional libraries. 
       (the word "supported", in this sense, means a compiler that runs on your
       HOST build platform, and generates output executable files for the TARGET
       platform.

It should be mentioned that you can execute the cross-compile build approach
even to generate binaries for the same platform as the host, where the HOST
compilerand the TARGET compiler are actually one and the same.


  +------------------------------------------------------+
  | B2. What needs to be built and executed on the HOST? |
  +------------------------------------------------------+

On the HOST, here are the mandatory things that have to be built.

    a) Using the HOST native compiler, build  HOST native utility makedefs

       Compile and link the following with these compiler switches:
         -DCROSSCOMPILE and -DCROSSCOMPILE_HOST
       from sources: util/makedefs.c, src/mdlib.c, src/monst.c, src/objects.c

    b) Execute HOST native makedefs utility, util/makedefs, as follows:
          util/makedefs -v
          util/makedefs -o
          util/makedefs -p
          util/makedefs -z
          util/makedefs -d
          util/makedefs -r
          util/makedefs -h
          util/makedefs -s

    c) Using the HOST native compiler, build these additional utilities if your
       target platform requires components that they produce. It is important
       to note that all of the required source files need to be compiled to
       native obj files for linking into the HOST-side utility. Some of the
       source files (src/monst.c, src/objects.c) were likely already compiled
       as native HOST-side obj files in order to build the native HOST utility
       'makedefs' HOST utility above, and you don't need to compile them again
       for use in linking other HOST utilities if the HOST-native obj files
       produced from them are still around.  

       NOTE: A few other source files (src/drawing.c, src/decl.c) that need to
       be compiled for native HOST utilities, also need to be compiled again
       later as TARGET obj files for linking into the TARGET executable. It is
       important to keep the compiled HOST-side obj files produced by the HOST
       native compiler, and the TARGET-side obj files produced by the
       cross-compiler separate and distinct from one another. That can be done
       either by naming the differing object files a little differently
       (perhaps with a suffix), or by placing the HOST-side obj files and the
       TARGET-side obj files into different folders during the build process.
       Whatever works best for your cross-compile.
       
          util/dlb

                   from sources: src/dlb.c, src/dlb_main.c, src/alloc.c,
                                 src/panic.c
                   purpose:      For packaging up many files that are
                                 required components of the TARGET game
                                 into a single nhdat or nhdat370 combined
                                 file

          util/uudecode

                   from sources: sys/share/uudecode.c
                   purpose:      convert some binary files, that are 
                                 distributed in the NetHack sources in
                                 uuencoded format, back into their
                                 original binary state
          util/tilemap

                   from sources: win/share/tilemap.c
                   purpose:      produce output file src/tile.c that is
                                 required for building TARGET packages with
                                 tile support

          util/tile2bmp

                   from sources: win/share/tile2bmp.c, win/share/tiletext.c,
                                 win/share/tilemap.c, src/drawing.c,
                                 src/decl.c, src/monst.c, src/objects.c
                   purpose:      <TBD>

          util/gif2txt

                   from sources: win/share/gifread.c, win/share/tiletext.c,
                                 win/share/tilemap.c, src/drawing.c,
                                 src/decl.c, src/monst.c, src/objects.c,
                                 src/alloc.c, src/panic.c
                   purpose:      <TBD>

          util/ppmwrite

                   from sources: win/share/ppmwrite.c, win/share/tiletext.c,
                                 win/share/tilemap.c, src/drawing.c,
                                 src/decl.c, src/monst.c, src/objects.c,
                                 src/alloc.c, src/panic.c
                   purpose:      <TBD>


  +--------------------------------------------+
  | B3. What needs to be built for the TARGET? |
  +--------------------------------------------+


For the TARGET side, here are the mandatory things that have to be built via
the HOST-executed cross-compiler that generates code for the TARGET platform.

Using the cross-compiler, build the following targets:

    a) NetHack sources (core is mandatory)

       With the cross-compiler and linker for the TARGET platform,
       cross-compile and link with these compiler switches:
         -DCROSSCOMPILE and -DCROSSCOMPILE_TARGET

       core sources (2019): src/allmain.c, src/apply.c, src/artifact.c,
                           src/attrib.c, src/ball.c, src/bones.c,
                           src/botl.c, src/cmd.c, src/dbridge.c, src/decl.c,
                           src/detect.c, src/dig.c, src/display.c, src/do.c,
                           src/do_name.c, src/do_wear.c, src/dog.c,
                           src/dogmove.c, src/dokick.c, src/dothrow.c,
                           src/drawing.c, src/dungeon.c, src/eat.c, src/end.c,
                           src/engrave.c, src/exper.c, src/explode.c,
                           src/extralev.c, src/files.c, src/fountain.c,
                           src/hack.c, src/hacklib.c, src/insight.c, src/invent.c,
                           src/isaac64.c, src/light.c, src/lock.c, src/mail.c,
                           src/makemon.c, src/mapglyph.c, src/mcastu.c,
                           src/mdlib.c, src/mhitm.c, src/mhitu.c, src/minion.c,
                           src/mklev.c, src/mkmap.c, src/mkmaze.c, src/mkobj.c,
                           src/mkroom.c, src/mon.c, src/mondata.c, src/monmove.c,
                           src/monst.c, src/mplayer.c, src/mthrowu.c, src/muse.c,
                           src/music.c, src/nhlsel.c, src/nhlua.c, src/nhlobj.c,
                           src/o_init.c, src/objects.c, src/objnam.c,
                           src/options.c, src/pager.c, src/pickup.c, src/pline.c,
                           src/polyself.c, src/potion.c, src/pray.c,
                           src/priest.c, src/quest.c, src/questpgr.c,
                           src/random.c, src/read.c, src/rect.c, src/region.c,
                           src/restore.c, src/rip.c, src/rnd.c, src/role.c,
                           src/rumors.c, src/save.c, src/sfstruct.c,
                           src/shk.c, src/shknam.c, src/sit.c, src/sounds.c,
                           src/sp_lev.c, src/spell.c, src/steal.c, src/steed.c,
                           src/sys.c, src/teleport.c, src/tile.c,
                           src/timeout.c, src/topten.c, src/track.c,
                           src/trap.c, src/u_init.c, src/uhitm.c, src/vault.c,
                           src/version.c, src/vis_tab.c, src/vision.c,
                           src/weapon.c, src/were.c, src/wield.c, src/windows.c,
                           src/wizard.c, src/worm.c, src/worn.c, src/write.c,
                           src/zap.c, sys/share/cppregex.cpp

       tty sources:        win/tty/getline.c, win/tty/termcap.c,
                           win/tty/topl.c, win/tty/wintty.c

       plus your platform-specific source files that contain main, typically
       *main.c, and unix support in *unix.c, tty support in *tty.c, and other
       system support in *sys.c as well as others sources pertaining to your
       specific target platform(s).

    b) Lua (mandatory in 3.7)

          lib/lua-5.3.5/src

                   from sources: lua.c, lapi.c, lauxlib.c, lbaselib.c, lcode.c,
                                 lcorolib.c, lctype.c, ldblib.c, ldebug.c,
                                 ldo.c, ldump.c, lfunc.c, lgc.c, linit.c,
                                 liolib.c, llex.c, lmathlib.c, lmem.c,
                                 loadlib.c, lobject.c, lopcodes.c,
                                 loslib.c, lparser.c, lstate.c, lstring.c,
                                 lstrlib.c, ltable.c, ltablib.c, ltm.c,
                                 lundump.c, lutf8lib.c, lvm.c, lzio.c,
                                 lbitlib.c
                   purpose:      links into the game executable to interpret
                                 lua level description files, lua dungeon
                                 description files, and a lua quest text file.

    d) recover (optional if desired/required; some targets have recover
       functionality built into NetHack itself)

    c) Additional optional library packages/obj files as required

          lib/pdcurses/...

                   from sources: addch.c, addchstr.c, addstr.c, attr.c, beep.c,
                                 bkgd.c, border.c, clear.c, color.c, delch.c,
                                 deleteln.c, getch.c, getstr.c, getyx.c,
                                 inch.c, inchstr.c, initscr.c, inopts.c,
                                 insch.c, insstr.c, instr.c, kernel.c,
                                 keyname.c, mouse.c, move.c, outopts.c,
                                 overlay.c, pad.c, panel.c, printw.c,
                                 refresh.c, scanw.c, scr_dump.c, scroll.c,
                                 slk.c, termattr.c, touch.c, util.c, window.c,
                                 debug.c, pdcclip.c, pdcdisp.c, pdcgetsc.c,
                                 pdckbd.c, pdcutil.c
                   purpose:      underlying curses platform support for some
                                 target platforms where inclusion of the
                                 NetHack curses window port in win/curses is
                                 desired

  +-------------------------+
  | B4. Case sample: msdos  |
  +-------------------------+

Cross-compiler used:     Andrew Wu's djgpp cross-compiler
Cross-compiler url:      https://github.com/andrewwutw/build-djgpp
Cross-compiler pre-built binary downloads:
    https://github.com/andrewwutw/build-djgpp/releases/download/v2.9/
    Mac OS X pre-built binary: djgpp-osx-gcc550.tar.bz2              (tested)
    Linux pre-built binary   : djgpp-linux64-gcc550.tar.bz2          (tested)
    mingw pre-built binary   : djgpp-mingw-gcc550-standalone.zip     (untested)

The msdos cross-compile for NetHack 3.7 uses two phases of compiles: 
Phase1 is the host-side prerequisite stuff that needs to be done first. 
Phase2 is the cross-compile pieces using the djgpp cross-compiler hosted on
Linux, Mac OS X, or Windows mingw.

First, on the host platform, you need to set up for a native Unix NetHack
build in the usual way. For example, on linux:
    cd sys/unix
    sh setup.sh hints/linux
    cd ../..
    make fetch-lua

Now, you could proceed to go ahead and issue
    make all
to build a native NetHack at that point if you wish, but it is not needed
for the msdos cross-compile. 

Instead, a test shell script has been put together that will next accomplish
each of the following tasks when it is executed. The shell script can be
invoked by:
    sh sys/msdos/msdos-cross-compile.sh
but before you do that, please read the paragraphs below.

The shell script is meant to accomplish the following things:

    Prep   : the script downloads the djgpp cross-compiler for the host
             platform into lib/djgpp (it doesn't install anything on the
             system, nor does it need to, it just downloads them into the
             identified directories), it downloads a copy of the msdos
             dos-extender into lib/djgpp/cwsdpmi for later packaging up with
             the msdos game, and it downloads pdcurses into lib/pdcurses
             for cross-compiling during the TARGET build.

             Be certain to ensure the right products are at the url's
             identified above *before* you execute the Case sample msdos
             cross-compile script. The correct products were at those url's
             at the time this was written in Dec 2019, but we don't assume
             any responsibility for what is at those url's now or in the
             future. You need to check before executing the script.

    Phase1 : the script uses the Makefile sys/msdos/Makefile1.cross 
             to complete the host-side build steps using the native gcc
             compiler for the host platform. During phase1 the host obj
             files are put in subfolder src/host_o to keep them separated
             and distinguishable from the target obj files that will be
             built in phase2.

    Phase2 : the script uses the Makefile sys/msdos/Makefile2.cross
             to complete the target-side build steps using the
             cross-compiler that was obtained during the prep step of the
             script described above. During phase2 the target obj files
             are put in src/msdos_o to keep them separated and
             distinguishable from the host obj files

    Package: the script then packages up the results that reside in
             msdos-binary into a zip file which it places in lib called
             nh370dos.zip.