Booting Fedora Linux on RISC-V QEMU emulator in 2022
I wanted to learn about RISCV this holiday break. You can use QEMU to emulate a RISC-V machine environment and you can run Fedora Linux on it. However, it was not straightforward. There are several tutorials online, but things seem to have changed and most of these didn’t successfully boot Linux for me. It took several days, but I finally managed to boot it. Here’s how I did it.
My goal
My goal was to log on to a Linux environment and issue ls command. As I don’t have a RISC-V hardware, I run it on QEMU.
My environment
It should be possible to do this on any Linux system, but my particular environment is:
- Fedora Linux 35 (host)
- QEMU 6.2.0
- Fedora Developer Rawhide 20211110 (for RISC-V)
- OpenSBI v1.0 (Mainline - github master as of 2022-01-03)
- u-boot 2022.01-rc4 (github master as of 2022-01-03)
Steps
QEMU
QEMU is a versatile emuation environment and it can emulate various CPUs and machines including RISC-V and x86.
For RISC-V emuation, you need qemu-system-riscv64.
To install a relatively new QEMU build, I used @copr:copr.fedorainfracloud.org:group_sig:virt-preview. I’ve followed the instrunction here.
On my system qemu-system-riscv64 is provided by qemu-system-riscv-core-6.2.0-1.fc35.x86_64.
$ qemu-system-riscv64 --version
QEMU emulator version 6.2.0 (qemu-6.2.0-1.fc35)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
RISC-V Cross compiler
You need a RISC-V Cross compiler so that you can build firmwares(OpenSBI and u-boot) that runs on the RISC-V emulator.
toolchains.bootlin.com provides a large number of ready-to-use cross-compilation toolchains.
I’ve downloaded one for [risc64-lp64d] architecture and [glibc] as libc.
I’ve extracted the .bz2 file to /home/gaku/tz/riscv64-lp64d--glibc--stable-2021.11-1/.
I’ve checked that RISC-V gcc cross compiler is there.
$ /home/gaku/tz/riscv64-lp64d--glibc--stable-2021.11-1/bin/riscv64-linux-gcc --version
riscv64-linux-gcc.br_real (Buildroot toolchains.bootlin.com-2021.11-1) 10.3.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$
u-boot
I’ve used the official u-boot (https://github.com/u-boot/u-boot).
$ git clone https://github.com/u-boot/u-boot
Make a small patch to include/configs/qemu-riscv.h to address kernel_comp_size and kernel_comp_addr_r issues (See troubleshooting section).
#define CONFIG_EXTRA_ENV_SETTINGS \
"fdt_high=0xffffffffffffffff\0" \
"initrd_high=0xffffffffffffffff\0" \
"kernel_addr_r=0x84000000\0" \
"fdt_addr_r=0x88000000\0" \
"scriptaddr=0x88100000\0" \
"pxefile_addr_r=0x88200000\0" \
"ramdisk_addr_r=0x88300000\0" \
"kernel_comp_size=0x4000000\0" \ <=== add this
"kernel_comp_addr_r=0x90000000\0" \ <=== add this
BOOTENV
#endif
I built u-boot using the cross compiler I’ve installed:
$ export PATH=/home/gaku/tz/riscv64-lp64d--glibc--stable-2021.11-1/bin:$PATH
$ export CROSS_COMPILE=riscv64-buildroot-linux-gnu-
$ make qemu-riscv64_smode_defconfig
$ make
This makes the build process to use riscv64-buildroot-linux-gnu-gcc compiler from /home/gaku/tz/riscv64-lp64d--glibc--stable-2021.11-1/bin directory, i.e. to compile u-boot with the RISC-V cross compiler.
U-boot is built with qemu-riscv64_smode_defconfig configuration.
S-Mode (supervisor mode) is a mode of RISC-V processor. It is a less powerful mode than M-Mode(machine mode). It looks like a few things will be done via OpenSBI instead of directly manupilating hardware.
There’s another u-boot config called qemu-riscv64_spl_defaconfig, but I am not using it now.
By making u-boot, you will get u-boot.bin under u-boot directory (/home/gaku/tz/u-boot/uboot.bin for me). We will use this with building OpenSBI next.
OpenSBI
OpenSBI is the first thing that runs on a RISC-V machine. It is a firmware and I guess it is similar to a BIOS of a PC.
I’ve cloned the repo from Github(riscv-software-src/opensbi).
You want to built it with the following commands. Same as u-boot, use the cross compiler.
export PATH=/home/gaku/tz/riscv64-lp64d--glibc--stable-2021.11-1/bin:$PATH
export CROSS_COMPILE=riscv64-buildroot-linux-gnu-
make PLATFORM=generic FW_PAYLOAD_PATH=/home/gaku/tz/u-boot/u-boot.bin
PLATFORM=generic specifies the hardware configuration for OpenSBI. There are a few other configurations under platform/ directory.
You can embed a payload to an OpenSBI firmware. We are putting the u-boot we built in the previous section using FW_PAYLOAD_PATH.
make will create platform/generic/firmware/fw_payload.elf and platform/generic/firmware/fw_payload.bin.
These are the firmware file (OpenSBI + u-boot) we use with QEMU.
Fedora disk image for RISC-V
It seems like Fedora Linux made the most progress on RISC-V support among various Linux distributions. However, online tutorials were written around 2019 or 2020 and are often outdated.
I wanted to use something newer. There’s a Fedora build system at fedora.riscv.rocks.
Pick the latest one (Fedora-Developer-Rawhide-20211110.n.0) and download Fedora-Developer-Rawhide-20211110.n.0-sda.raw.xz. It is a disk image of Fedora Rawhide snapshot on 20211110. There’s ‘Developer’ version which contains more packages.
Preparing for the first boot
I’ve create a directory to have all the files I need.
$ mkdir -p ~/tz/fedora-riscv
$ cd ~/tz/fedora-riscv
$ mv /home/gaku/Downloads/Fedora-Developer-Rawhide-20211110.n.0-sda.raw .
$ cp /home/gaku/tz/opensbi/platform/generic/firmware/fw_payload.elf
$ ls
Fedora-Developer-Rawhide-20211110.n.0-sda.raw
fw_payload.elf
There are some discrepancies between the u-boot we compiled and the Fedora RISC-V provides, and it doesn’t work as is. I need to update /boot/extlinux/extlinux.conf file in the raw image using guestfish command.
$ guestfish -a Fedora-Developer-Rawhide-20211110.n.0-sda.raw -i
Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.
Type: ‘help’ for help on commands
‘man’ to read the manual
‘quit’ to quit the shell
Operating system: Fedora 33 (Rawhide)
/dev/sda2 mounted on /
/dev/sda1 mounted on /boot
><fs>edit /boot/extlinux/extlinux.conf
Change the line with appappend to append.
# extlinux.conf generated by appliance-creator
ui menu.c32
menu autoboot Welcome to Fedora-Developer-Rawhide-20211110.n.0. Automatic boot in # second{,s}. Press a key for options.
menu title Fedora-Developer-Rawhide-20211110.n.0 Boot Options.
menu hidden
timeout 20
totaltimeout 600
label Fedora-Developer-Rawhide-20211110.n.0 (5.14.16-101.0.riscv64.fc33.riscv64)
kernel /vmlinuz-5.14.16-101.0.riscv64.fc33.riscv64
append ro root=UUID=1abf5c6d-3827-4beb-8b8b-3d39e5e14d6d rhgb LANG=en_US.UTF-8
initrd /initramfs-5.14.16-101.0.riscv64.fc33.riscv64.img
and quit to exit from guestfish.
Now create a shell script to start QEMU.
#!/bin/sh
qemu-system-riscv64 \
-nographic \
-machine virt \
-smp 8 \
-m 2G \
-bios fw_payload.elf \
-device virtio-blk-device,drive=hd0 \
-drive file=Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,id=hd0 \
-device virtio-net-device,netdev=usernet \
-netdev user,id=usernet,hostfwd=tcp::10000-:22
Then, run the command.
$ sh run-20220102.sh
OpenSBI v1.0
____ _____ ____ _____
/ __ \ / ____| _ \_ _|
| | | |_ __ ___ _ __ | (___ | |_) || |
| | | | '_ \ / _ \ '_ \ \___ \| _ < | |
| |__| | |_) | __/ | | |____) | |_) || |_
\____/| .__/ \___|_| |_|_____/|____/_____|
| |
|_|
Platform Name : riscv-virtio,qemu
Platform Features : medeleg
Platform HART Count : 8
Platform IPI Device : aclint-mswi
Platform Timer Device : aclint-mtimer @ 10000000Hz
Platform Console Device : uart8250
Platform HSM Device : ---
Platform Reboot Device : sifive_test
Platform Shutdown Device : sifive_test
Firmware Base : 0x80000000
Firmware Size : 308 KB
Runtime SBI Version : 0.3
Domain0 Name : root
Domain0 Boot HART : 3
Domain0 HARTs : 0*,1*,2*,3*,4*,5*,6*,7*
Domain0 Region00 : 0x0000000002000000-0x000000000200ffff (I)
Domain0 Region01 : 0x0000000080000000-0x000000008007ffff ()
Domain0 Region02 : 0x0000000000000000-0xffffffffffffffff (R,W,X)
Domain0 Next Address : 0x0000000080200000
Domain0 Next Arg1 : 0x0000000082200000
Domain0 Next Mode : S-mode
Domain0 SysReset : yes
Boot HART ID : 3
Boot HART Domain : root
Boot HART ISA : rv64imafdcsu
Boot HART Features : scounteren,mcounteren,time
Boot HART PMP Count : 16
Boot HART PMP Granularity : 4
Boot HART PMP Address Bits: 54
Boot HART MHPM Count : 0
Boot HART MIDELEG : 0x0000000000000222
Boot HART MEDELEG : 0x000000000000b109
U-Boot 2022.01-rc4-00030-gb3f84a939f-dirty (Jan 02 2022 - 23:40:19 -0800)
CPU: rv64imafdcsu
Model: riscv-virtio,qemu
DRAM: 2 GiB
Flash: 32 MiB
Loading Environment from nowhere... OK
In: uart@10000000
Out: uart@10000000
Err: uart@10000000
Net: eth0: virtio-net#1
Hit any key to stop autoboot:
Device 0: QEMU VirtIO Block Device
Type: Hard Disk
Capacity: 9192.0 MB = 8.9 GB (18825216 x 512)
... is now current device
Scanning virtio 0:1...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
gaku - Ignoring unknown command: ui
Ignoring malformed menu command: autoboot
Ignoring malformed menu command: hidden
gaku - Ignoring unknown command: totaltimeout
1: Fedora-Developer-Rawhide-20211110.n.0 (5.14.16-101.0.riscv64.fc33.riscv64)
Retrieving file: /initramfs-5.14.16-101.0.riscv64.fc33.riscv64.img
Retrieving file: /vmlinuz-5.14.16-101.0.riscv64.fc33.riscv64
append: ro root=UUID=1abf5c6d-3827-4beb-8b8b-3d39e5e14d6d rhgb LANG=en_US.UTF-8
Uncompressing Kernel Image
Moving Image from 0x84000000 to 0x80200000, end=82155000
## Flattened Device Tree blob at ff73c220
Booting using the fdt blob at 0xff73c220
Using Device Tree in place at 00000000ff73c220, end 00000000ff740e1d
Starting kernel ...
[ 0.000000] Linux version 5.14.16-101.0.riscv64.fc33.riscv64 (mockbuild@0eb270f0aea2488598ed23877b4f9789) (gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1), GNU ld version 2.35-18.fc33) #1 SMP Tue Nov 9 05:04:52 EST 2021
...
Then, Linux is booting up.
Welcome to the Fedora/RISC-V disk image
https://fedoraproject.org/wiki/Architectures/RISC-V
Build date: Wed Nov 10 20:06:54 UTC 2021
Kernel 5.14.16-101.0.riscv64.fc33.riscv64 on an riscv64 (ttyS0)
The root password is 'fedora_rocks!'.
root password logins are disabled in SSH starting Fedora 31.
User 'riscv' with password 'fedora_rocks!' in 'wheel' group is provided.
To install new packages use 'dnf install ...'
To upgrade disk image use 'dnf upgrade --best'
If DNS isn’t working, try editing ‘/etc/yum.repos.d/fedora-riscv.repo’.
For updates and latest information read:
https://fedoraproject.org/wiki/Architectures/RISC-V
Fedora/RISC-V
-------------
Koji: http://fedora.riscv.rocks/koji/
SCM: http://fedora.riscv.rocks:3000/
Distribution rep.: http://fedora.riscv.rocks/repos-dist/
Koji internal rep.: http://fedora.riscv.rocks/repos/
fedora-riscv login: riscv <== type this
Password: <== Type password 'fedora_rocks!'
Last login: Tue Dec 28 20:45:18 on ttyS0
[riscv@fedora-riscv ~]$ cd /
[riscv@fedora-riscv /]$ ls
bin dev home lib64 media opt root sbin sys usr
boot etc lib lost+found mnt proc run srv tmp var
[riscv@fedora-riscv /]$ sudo bash
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
[sudo] password for riscv:
[root@fedora-riscv /]# poweroff
...
[ 302.289370] systemd-shutdown[1]: Powering off.
[ 302.293821] Flash device refused suspend due to active operation (state 20)
[ 302.294401] Flash device refused suspend due to active operation (state 20)
[ 302.296214] reboot: Power down
$
I’ve completed my goal!
Troubleshooting
Fedora Linux doesn’t boot. Seeing Ignore unknown command: initrd.
Device 0: QEMU VirtIO Block Device
Type: Hard Disk
Capacity: 9192.0 MB = 8.9 GB (18825216 x 512)
... is now current device
Scanning virtio 0:1...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
Ignoring unknown command: ui
Ignoring malformed menu command: autoboot
Ignoring malformed menu command: hidden
Ignoring unknown command: totaltimeout
Ignoring unknown command: addappend
Ignoring unknown command: initrd
1: Fedora-Developer-Rawhide-20211110.n.0 (5.14.16-101.0.riscv64.fc33.riscv64)
Retrieving file: /vmlinuz-5.14.16-101.0.riscv64.fc33.riscv64
kernel_comp_addr_r or kernel_comp_size is not provided!
SCRIPT FAILED: continuing...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Scanning disk virtio-blk#0...
Found 3 disks
No EFI system partition
BootOrder not defined
EFI boot manager: Cannot load any image
scanning bus for devices...
Device 0: unknown device
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (2 ms)
Using virtio-net#1 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename 'boot.scr.uimg'.
Load address: 0x88100000
Loading: *
TFTP error: 'Access violation' (2)
Not retrying...
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (1 ms)
Using virtio-net#1 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename 'boot.scr.uimg'.
Load address: 0x84000000
Loading: *
TFTP error: 'Access violation' (2)
Not retrying...
=>
This happened because extlinux.conf’s content was not compatible with what u-boot expected. Fedora’s nightly built uses addappend in extlinux.conf but that is not supported. Instead use append.
You may still have problem.
kernel_comp_addr_r or kernel_comp_size is not provided!
Both kernel_comp_addr_r and kernel_comp_size u-boot environment variables were introduced around March 2020. You need these variables defined to boot a Linux Kernel.
Patch u-boot/include/configs/qemu-riscv.h:
#define CONFIG_EXTRA_ENV_SETTINGS \
"fdt_high=0xffffffffffffffff\0" \
"initrd_high=0xffffffffffffffff\0" \
"kernel_addr_r=0x84000000\0" \
"fdt_addr_r=0x88000000\0" \
"scriptaddr=0x88100000\0" \
"pxefile_addr_r=0x88200000\0" \
"ramdisk_addr_r=0x88300000\0" \
"kernel_comp_size=0x4000000\0" \ <=== add this
"kernel_comp_addr_r=0x90000000\0" \ <=== add this
BOOTENV
#endif
I am not 100% sure about these values, but I refereed sifive-unmatched.h’s config and it was close enough, and it worked.