Filesystems and startup

The Circle kernel is based on a standard openwrt build, using the Atheros AP123/TP-Link TL-MR3420-v2 platform.

The boot flash chip is 8 megabytes (SPI), and contains u-boot, the kernel, the startup root filesystem (read-only), and a read-write jffs2 rootfs overlay partition (which is mounted as an overlay on top of the startup root filesystem – effectively making “/” read-write.

In addition, there’s an embedded “eMMC” flash chip, which appears on the USB bus as “/dev/sda”.  The device claims to be 8 gigabytes, but I’ve found that it’s only 1 gigabyte on my particular Circle device (doing a “dd” after the 1 gigabyte mark results in the eMMC chip “resetting” and unplugging/replugging on the USB bus.

Here’s the kernel startup messages showing the SPI flash partitions:

[ 0.700000] 5 tp-link partitions found on MTD device spi0.0
[ 0.700000] Creating 5 MTD partitions on "spi0.0":
[ 0.710000] 0x000000000000-0x000000020000 : "u-boot"
[ 0.720000] 0x000000020000-0x00000012e5fc : "kernel"
[ 0.720000] mtd: partition "kernel" must either start or end on erase block boundary or be smaller than an erase block -- forcing read-only
[ 0.740000] 0x00000012e5fc-0x0000007f0000 : "rootfs"
[ 0.740000] mtd: partition "rootfs" must either start or end on erase block boundary or be smaller than an erase block -- forcing read-only
[ 0.760000] mtd: device 2 (rootfs) set to be root filesystem
[ 0.760000] 1 squashfs-split partitions found on MTD device rootfs
[ 0.770000] 0x0000004d0000-0x0000007f0000 : "rootfs_data"
[ 0.780000] 0x0000007f0000-0x000000800000 : "art"
[ 0.780000] 0x000000020000-0x0000007f0000 : "firmware"

“rootfs” is the read-only startup root filesystem (initramfs, squashfs).  “rootfs_data” is the jffs2 read-write overlay.  “art” contains Atheros radio configuration data.  “firmware” is a pseudo-partition that maps to the entire 8-megabyte flash chip.

Here’s the output of the “mount” command after startup:

rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
sys on /sys type sysfs (rw,relatime)
proc on /proc type proc (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mtdblock3 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlayfs (rw,noatime,lowerdir=/,upperdir=/overlay)
tmpfs on /dev type tmpfs (rw,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
/dev/sda3 on /mnt type ext4 (rw,noatime,nodiratime)

As you can see, “/dev/sda3” (the eMMC flash) is mounted on “/mnt” (ext4).  Not shown above: “/dev/sda1” is a swap filesystem, and “/dev/sda2” is also ext4 – seems to be used to store a firmware image for a firmware recovery mode (in case of problems in the future).

Modifying /etc/passwd and /etc/shadow:

As I said, the jffs2 “rootfs_data” partition is mounted as an overlay onto “/”.  If you want to modify (for example) the “/etc/passwd” and “/etc/shadow” files, you can TFTP-boot an openwrt kernel (configured for the TP-Link TL-MR3420-v2), login, mount the jffs2 partition, then create  your own “/etc/passwd” and “/etc/shadow”.

NOTE: there is another method (using a modified firmware update) to modifying the passwd/shadow files (doesn’t require serial port or TFTP-booting a custom kernel).  I will document that in a separate post later.

Here’s what I did (I added a “dummy” user (with password “dummy”) to the system.  WARNING: if you screw-up the /etc/passwd and shadow files, it could cause the Circle device to fail to boot-up.  I felt safe doing this because I could always TFTP-boot my openwrt kernel using u-boot (so I could always restore the original passwd/shadow files.

NOTE: I first created “passwd.txt” and “shadow.txt” (starting with the “passwd/shadow” I had found in the main initramfs read-only rootfs), confirmed they were OK, then renamed them to “passwd” and “shadow”:

root@(none):/# cat /proc/mtd
dev: size erasesize name
mtd0: 00020000 00010000 "u-boot"
mtd1: 0010e5fc 00010000 "kernel"
mtd2: 006c1a04 00010000 "rootfs"
mtd3: 00320000 00010000 "rootfs_data"
mtd4: 00010000 00010000 "art"
mtd5: 007d0000 00010000 "firmware"
root@(none):/# mount -t jffs2 /dev/mtdblock3 /tmp/mnt
[ 28.240000] jffs2: notice: (324) jffs2_build_xattr_subsystem: complete building xattr subsystem, 1 of xdatum (1 unchecked, 0 orphan) and 19 of xref (0 dead, 8 orphan) found.
root@(none):/# cd /tmp/mnt
root@(none):/tmp/mnt# ls
etc lib usr
root@(none):/tmp/mnt# cd etc
root@(none):/tmp/mnt/etc# ls
circle config dropbear hosts modules.d uci-defaults
# NOTE: I did this "cat > passwd.txt", then pasted my modified data to the file
root@(none):/tmp/mnt/etc# cat > passwd.txt
# NOTE: I did this "cat > shadow.txt", then pasted my modified data to the file
root@(none):/tmp/mnt/etc# cat > shadow.txt
root@(none):/tmp/mnt/etc# chmod og-r shadow.txt
root@(none):/tmp/mnt/etc# chmod a+rw passwd.txt
root@(none):/tmp/mnt/etc# ls -la passwd.txt shadow.txt
-rw-rw-rw- 1 root root 222 Jan 1 00:01 passwd.txt
-rw------- 1 root root 213 Jan 1 00:01 shadow.txt
root@(none):/tmp/mnt/etc# mv passwd.txt passwd
root@(none):/tmp/mnt/etc# mv shadow.txt shadow
root@(none):/tmp/mnt/etc# cd
root@(none):~# sync
root@(none):~# umount /tmp/mnt
root@(none):~# reboot


Starting Circle firmware:

As I said, there’s a read-only root filesystem used for startup, and the eMMC is mounted to “/mnt”.  The eMMC contains the majority of the Circle firmware – where all of their applications/etc reside.

Here’s the “/etc/rc.local” (on the read-only root filesystem) showing where they mount the eMMC and start the Circle firmware:


fatalError() {
 echo "Fatal Error: $1"
 echo "fastblink" > /tmp/blueled

#Main Entry Point

exec 1>/dev/console
exec 2>/dev/console

echo "Starting rc.local"

echo "leds-gpio" > /sys/bus/platform/drivers/leds-gpio/unbind
echo "slowblink" > /tmp/blueled

ifconfig eth0

#make sure SD card is detected
if [ ! -e /dev/sda ] 
 sleep 5;
 if [ ! -e /dev/sda ] 
 fatalError "No SD card detected".

#check whether SD card is partitioned correctly
[ -e /dev/sda1 ] && [ -e /dev/sda2 ] && [ -e /dev/sda3 ] && sdready="yes"
[ "$sdready" = "yes" ] || fatalError "SD card not partitioned correctly"

#SD card is formatted properly, fsck it and mount it
e2fsck -y /dev/sda3
mount -t ext4 -o rw,noatime,nodiratime /dev/sda3 /mnt || fatalError "Mounting SD card failed"

failsafe | grep 'failsafe packet received' && {
 echo "Entering failsafe mode."
echo "Done with rc.local"

if [ -f /mnt/shares/usr/bin/startcircle ] 
 source /mnt/shares/usr/bin/startcircle
 fatalError "Error: did not find startcircle file"

And, here’s the “startcircle” script from the eMMC:

#final boot-up script to start circle functionality

export PATH=$PATH:$DIR
echo 3 > /proc/sys/vm/drop_caches

cp $DIR/myreboot /tmp/

#hardcode timezone for now
echo PST8PDT,M3.2.0,M11.1.0 > /etc/TZ

#set date to 2000 so we can detect when we get ntp time
date -s 2015.09.11-00:00

iw dev wlan0 interface add apcli0 type station 
iw dev wlan0 interface add ra0 type __ap
export WAN="eth0"
ln -s /sys/class/leds/tp-link\:green\:3g/brightness /tmp/blueled
MAC=`ifconfig apcli0 | awk '/HWaddr/{print $5;exit;}'`
echo "$MAC" > /tmp/MAC;
ETHMAC=`ifconfig eth0 | awk '/HWaddr/{print $5;exit;}'`
ip link set ra0 address $ETHMAC
grep "^8C:E2:DA" /tmp/MAC || {
 if [ -s /etc/MAC ] ; then
 cp -f /etc/MAC /tmp/MAC;
 MAC=`cat /etc/MAC`;
 ip link set apcli0 address $MAC
 echo "8C:E2:DA:F0:F0:01" > /tmp/MAC;
 ip link set apcli0 address 8C:E2:DA:F0:F0:01
 ip link set ra0 address 8C:E2:DA:F0:F0:00
 ifconfig eth0 down
 ip link set eth0 address 8C:E2:DA:F0:F0:00

#check base firmware files
[ -f $DIR/ledd ] && { diff $DIR/ledd /usr/bin/ledd > /dev/null || { cp -f $DIR/ledd /usr/bin/ledd; chmod +x /usr/bin/ledd; killall ledd; sleep 1; ledd & } }
#the running ledd may be the ledd in the ROM, before JFFS2 was loaded
diff /usr/bin/ledd /rom/usr/bin/ledd > /dev/null || { killall ledd; sleep 1; ledd & }
diff -r $DIR/scripts/circle /etc/circle > /dev/null || { cp -f $DIR/scripts/circle/* /etc/circle/; chmod +x /etc/circle/*; }

#start mycircle AP
ifconfig eth0
macid=`awk -F: '{print $5 $6}' /tmp/MAC`
cp -f $DIR/scripts/hostapd.conf /tmp/;
grep "Circle-$macid" /tmp/hostapd.conf > /dev/null || sed -i "s/^ssid=.*/ssid=Circle-$macid/g" /tmp/hostapd.conf
grep "MyCircle-$macid" /tmp/hostapd.conf > /dev/null && sed -i "s/^ssid=.*/ssid=Circle-$macid/g" /tmp/hostapd.conf
mkdir -p /var/lib/misc/

#PAUSE chain in iptables
iptables -N PAUSE
iptables -I FORWARD -j PAUSE
iptables -t raw -N RAWPAUSE
iptables -t raw -A PREROUTING -j RAWPAUSE
#Minecraft Pocket Edition Chain
iptables -t raw -N MCPE
iptables -t raw -A PREROUTING -j MCPE

ln -s /usr/bin/wget /tmp/

if [ ! -s $DIR/configure.xml ]; then
 if [ -s $DIR/configure.xml.backup ]; then
 cp $DIR/configure.xml.backup $DIR/configure.xml
 cp $DIR/configure-default.xml $DIR/configure.xml

ifconfig apcli0 up
iptables -t nat -A PREROUTING -i apcli0 -p udp --dport 53 -j REDIRECT

#start cron job
$DIR/tinycron 3600 $DIR/

#install missing packages 
if [ ! -s /usr/sbin/ipset ]; then
 opkg install $DIR/pkgs/libmnl_1.0.3-1_ar71xx.ipk
 opkg install $DIR/pkgs/kmod-nfnetlink_3.10.49-1_ar71xx.ipk
 opkg install $DIR/pkgs/kmod-ipt-ipset_3.10.49+6.20.1-1_ar71xx.ipk
 opkg install $DIR/pkgs/ipset_6.20.1-1_ar71xx.ipk
 modprobe ip_set
 modprobe ip_set_hash_ip
 modprobe ip_set_hash_net
 modprobe xt_set

#ipset init
ipset create vpns hash:ip
ipset create minecraft hash:ip
ipset create games hash:ip
ipset create games_net hash:net
while read p ; do 
 if [ ${p:0:1} != "#" ]; then
 ipset add games_net $p; 
done < $DIR/scripts/iprange_games.txt
ipset -file $DIR/scripts/ -exist restore

#start services
cp -a $DIR/service /var
runsvdir /var/service >/dev/null 2>/dev/null &

#dhcp client script will start the rest of the processes

echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/apcli0/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/$WAN/send_redirects

echo 1 > /proc/sys/net/ipv4/conf/apcli0/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/apcli0/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/$WAN/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/$WAN/arp_announce

echo 8 > /proc/sys/net/ipv4/tcp_retries2

sysctl -w net.ipv4.conf.apcli0.send_redirects=0
sysctl -w net.ipv4.conf.all.send_redirects=0

Finally, here’s the output of “find” showing all of the files on the eMMC /dev/sda3 partition:




Leave a Reply