andreiw / ppc64le_hello

ppc64le_hello - ``Hello World'' for a Little-Endian OpenPower world
===================================================================

This is an example of running a 64-bit little-endian payload using
skiboot (https://github.com/open-power/skiboot) or another OPAL
firmware.

It shows the bare minimum needed to start writing
a little-endian kernel for the OpenPower ecosystem. Whereas
skiboot/test/hello_world is big-endian, this shows how
to make OPAL calls from LE as well as other minutiae like
setting up a basic C environment.

What you get:
- 64-bit real-mode HV LE operation
- logging via sim inteface (mambo_write)
- logging via OPAL firmware (opal_write)
- calling C code, stack/BSS/linkage setup/TOC
- calling BE code from LE
- FDT parsing, dumping FDT
- Taking and returning from exceptions, handling
  unrecoverable/nested exceptions.
- Timebase (i.e. the "timestamp counter"), decrementer
  and hypervisor decrementer manipulation with some basic timer
  support (done for periodic callbacks into OPAL).
- Running at HV alias addresses (loaded at 0x00000000200XXXXX,
  linked at 0x80000000200XXXXX). The idea being that the code
  will access physical RAM and its own data structures solely
  using the HV addresses.
- SLB setup: demonstrates 1T segments with 4K base page
  and 16M base page size. One segment (slot = 0) is used
  to back the HV alias addresses with 16M pages. Another
  segment maps EA to VA 1:1 using 4K pages.
- Very basic HTAB setup. Mapping and unmapping for pages
  in the 4K and 16M segments, supporting MPSS (16M pages
  in the 4K segment). No secondary PTEG. No eviction support.
  Not SMP safe. Any access within the HV alias addresses
  get mapped in. Any faults to other unmapped locations are
  crashes, as addresses below 0x8000000000000000 should only
  be explicit maps.
- Taking exception vectors with MMU on at the alternate
  vector location (AIL) 0xc000000000004000.
- Running unpriviledged code.
- Running VM supervisor real mode code with VRMA

Hopefully the code is clear enough and not too messy. I'm not
trying hard enough ;-(. I think the only code that has had any
thought put into it is the exception vectors, and other places
look especially crappy because I wanted to avoid writing an
actual kernel.

Building
========

You will need a LE 64-bit toolchain to build (i.e. ppc64le-linux).
A good source of toolchains is kernel.org. For building skiboot
(for testing) you will also need the BE 64-bit toolchain
(i.e. powerpc64-linux).

Skiboot is pulled-in as a submodule. 

You will need the POWER8 Functional Simulator set up or Benjamin
Herrenschmidt's PowerNV QEMU tree.

Good directions for POWER8 Functional Simulator:
https://www.flamingspork.com/blog/2014/12/03/running-skiboot-opal-on-the-power8-simulator/

Good directions for QEMU/PowerNV:
https://www.flamingspork.com/blog/2015/08/28/running-opal-in-qemu-the-powernv-platform/

$ make

For QEMU or real hardware:

$ make CONFIG_NOSIM=1

Running
=======

$ make test

This will run the simulator and pop-up an xterm for OPAL console 0. If you
don't like xterm, feel free to change it in ppc64le_hello.tcl.

For QEMU/PowerNV, something like following should do. Remember that skiboot.lid must be in $(CWD).

$  cd skiboot; ~/src/qemu/ppc64-softmmu/qemu-system-ppc64 -m 4G -M powernv -kernel ../ppc64le_hello

You're looking for output like -
...
20295980: (20295980): [20289411000,5] INIT: Starting kernel at 0x20010000, fdt at 0x303039f0 (size 0x13b4)
20296509: (20296507): Hello Mambo!
20297324: (20297322): Hello OPAL!
20297989: (20297987): _start = 0x8000000020010000
20317249: (20316639): _bss   = 0x800000002001A948
20336509: (20335291): _stack = 0x800000002001B000
20355769: (20353943): _end   = 0x800000002001C000
20375038: (20372604): KPCR   = 0x800000002001A9E8
20394308: (20391266): TOC    = 0x8000000020022900
20413578: (20409928): OPAL   = 0x30000000
20427317: (20423363): FDT    = 0x303039F0
20538569: (20534311): SLB size = 0x20
20582102: (20577768): TB freq = 512000000
20600359: (20595683): [20595886000,3] OPAL: Trying a CPU re-init with flags: 0x2
20609301: (20604625): [20601599000,3] SLW: Not found on chip 0
20613538: (20608862): [20609554000,3] ELOG: Error getting buffer to log error
20614306: (20609630): OPAL claims no HILE supported, pretend to know better...
20651344: (20646668): Unrecoverable exception stack top @ 0x800000002001CEC8
20691374: (20686085): HTAB (2048 ptegs, mask 0x7FF, size 0x40000) @ 0x8000000020040000
20866415: (20860060): SLB entries:
20875356: (20868963): 0: E 0x8000000008000000 V 0x4000000001000500
20906511: (20898864): 1: E 0x8000000 V 0x4000000000000400
20931456: (20922897): 2: E 0xC000000008000000 V 0x4000000002000400

21151537: (21141251): Pick your poison:
21163581: (21153295): Choices: (MMU = disabled):
21178712: (21338426):    (q) poweroff
21180946: (21170660):    (d) 5s delay
21191199: (21180913):    (D) toggle 5s timer
21205925: (21195639):    (e) test exception
21220012: (21209726):    (n) test nested exception
21238572: (21228286):    (f) dump FDT
21248825: (21238539):    (M) enable MMU
21260356: (21250070):    (m) disable MMU
21272526: (21262240):    (t) test MMU
21282779: (21272493):    (T) test MMU 16mb pages
21300061: (21289775):    (u) test non-priviledged code
21300061: (21299783):    (U) test VM real-mode code
21321177: (21310891):    (H) enable HV dec
21334625: (21324339):    (h) disable HV dec
21338435: (21324339):    (I) run initrd


You can interact with this:
---------------------------
(q) will shut down the sim using the attn instruction.
(d) poll the timebase registers until 5s of real time pass.
(D) toggles a periodic timer callback every 5s.
(e) tests triggering a system call with parameter 0xfeed. The system
    call handler will return back parameter << 16 | 0xface, i.e. 0xfeedface.
(n) tests triggering an exception from an exception by triggering
    a system call with parameter 0xdead. The exc.c logic keys off the parameter
    to trigger another exception, expectedly crashing the system, thus testing
    that the exc.c code can distinguish between recoverable and non-recoverable
    exceptions and handle them appropriately.
(f) dumps the FDT passed up by skiboot. I don't try very hard. This code needs
    improvement and its just glue around parts of libfdt.
(M) enables running with MSR.IR and MSR.DR. HTAB, memory between &_start
    and &_end and the AIL (alternate) vectors are mapped. When MMU is on,
    exception vectors are taken at EA 0xc000000000004000 with MMU on.
(m) disables MSR.IR and MSR.DR, disabling MMU. Exception vectors are taken with
    MMU off at RA 0x0.
(t) simple test for MMU code, mapping the same EA to different RAs and verifying
    that the 4K mapping worked.
(T) simple 16MB page size mapping test.
(u) shows running unpriviledged code. There's no scheduler, process/thread
    support so the hand-off between code is messy. We enter user code via
    syscall, saving off the exception frame for a subsequent return from user
    code, again via syscall.
(U) similar to (u), but shows running code in non-HV supervisor mode in
    real mode, by using the VRMA/VPM features of HV mode.
(H) enables HV decrementer. You will start seeing hypervisor decrementer
    exceptions.
(h) disable HV decrementer. You will no longer see any HV decrementer
    exceptions.
(I) execute LE binary (not ELF) code stored in initrd. The first uint64_t
    may either be code itself, a pointer to code or a pointer to a
    ABIv2 function descriptor.

You should be able to run ppc64le_hello on a real Power8 machine via skiboot,
but you will need to build with CONFIG_NOSIM=1. It has been successfully done
before, but that was before I wrote most of the MMU code ;-). See Daniel
Axtens' blog post:

http://sthbrx.github.io/blog/2015/06/03/ppc64le-hello-on-real-hardware/

Issues
======

Q: Delay test seems to run longer or shorter than 5s wall clock.
A: From benh: "The way mambo (aka systemsim) works, there is no expectation
               that the simulated timebase has any relationship whatsoever
               to wall clock outside of sim. All you can do really is play
               with the ratio emulated CPU cycles / TB cycles."
   ...change processor/cpu_frequency and processor/timebase_frequency
   inside ppc64le_hello.tcl. On my machine (i5-2410M) the TB should run
   about 1000 times faster than clock.

Q: Something doesn't work on real hardware!
A: Please file an issue and let me know! Daniel Axtens identified a number
   of issues which I believe to have resolved, but I only get to play in
   a simulator.

Q: Something doesn't work on QEMU!
A: 16MB (MPSS) pages don't seem to work as of 01-Oct-2015.

TODO
====

- Maybe show more callthru sim facilities? Like file I/O...
- Spinlocks
- SMP

Resources
=========

Useful stuff:
- PowerISA_V2.07_PUBLIC.pdf             - Book 3S
- ABI53BitOpenPOWER_21July2014_pub.pdf  - ELFv2 ABI used for PPC64LE
- P8_um_external_v1.1_2015JAN29_pub.pdf - POWER8 User Manual

Contact Info
============

Andrei Warkentin ([email protected]).