www.dmdtech.org

Introduction / Blah Blah (skip if you want)

Verizon, now Frontier (in my area) rolled out FIOS here over 15 years ago. Since then, it’s been a challenge with battery backup because the ONT is almost always installed at the phone company’s demarcation which is the garage or somewhere else inconvenient.

This is in comparison with the cable company which will usually pay the installer to run the coax to whichever room you want as part of a standard installation.

The garage is hot and not a great place for a UPS because of hot Texas summers that bake the insides of our garage. In the past, I’ve run lengths of extension cord through the attic and into the “computer room” to provide UPS power, but this is not ideal as it violates electrical code.

Extension cords are not allowed to be installed through ceilings, crawlspaces, walls, that sort of thing. The risk of a fire is likely remote if the only thing you are powering is an ONT due to the tiny current draw but it just did not feel right to run that much 120v extension through the attic and through two ceilings.

In my case, I recently moved the “computer room” even further away which was going to force me to either daisy-chain extension cords, put a UPS in a hot garage, or figure out how to steal PoE power from the CAT-6 line that I was already running.

Equipment List

You’ll need at a bare minimum:

• PoE switch or 48v PoE injector that is UPS protected (I am using my existing 48 port Cisco 3750X PoE switch)
• PoE splitter so you can split the power from the data at the ONT (I am using this Linovision PoE splitter /w selectable voltages)
• 5.5mm DC female plug if you are using the Linovision splitter.
• 4.5mm DC female plug for the auxiliary input port of the ONT’s power brick
• A sliver of aluminum foil (!)

I have about 30 years worth of electronics including power bricks, power leads, pigtails, electronics doo-dads, and that sort of thing so I was able to scavenge the needed plug ends from my collection and build a power cable.

Foil Party Trick

By using the aux input port on your ONT’s power brick (in my case, it’s a CyberPower ca25u16v2 power brick), you are basically supplying a secondary source of power that goes to the ONT in the event of a power failure.

Unfortunately, these ONTs will prioritize voice over data during a power outage. During a power outage, you get something like 10 to 15 minutes of data before the data is shut off.

To bypass this restriction, you need to short two pins together to fool the ONT into thinking it is getting full time primary power. Please see the link below.

It’s literally just a sliver of foil that you can jam into the connector. Once you do this, the ONT will accept secondary power without shutting off data after 10 minutes.

I had to do this twice; the first time the foil did not make good contact. The second time proved successful.

You will know if this is NOT successful if the power LED on the ONT flashes when main power is unplugged. It will continue to work but for a limited amount of time before shutting off data. If the power LED remains solid (give it a good 15 seconds to be sure) then you successfully fooled it with the foil bridge.

https://www.instructables.com/I-211M-L-ONT-Enable-Data-While-on-Battery-Power

Selecting The Right Voltage (Voltage Warning)

Depending on your ONT, it may accept as little as 12V or it may require 16V. If you are using a PoE splitter with selectable voltage, you probably don’t have an option for 16V. Pay attention to the data label on the back.

In my case, I have an FRG222 that specifies 16V on its data label. I tried powering it with 12V and the ONT would not take it successfully. Lacking a 16V option on my splitter, I tried it with 18V (thinking that 18V is only a hair more than 16V and the onboard electronics could deal with it). 18V worked fine for me. No ill effects observed.

Do this at your own risk. Try with 12V first before bumping it up to 18V. If you can find a PoE splitter online that supplies 16V, please let us know! The Linovision /w 18v is the closest I could find.

Final Result, Other Methods

The great thing about this method is that it is completely reversible and doesn’t require you butcher the existing power cable.

Other methods I’ve seen discussed involve butchering the proprietary power cable to supply the PoE voltage. That works but you got some ‘splaining to do if you ever have to have a tech visit. This way I can just hide all of my extra “crap” if a tech ever needs to visit for some reason and it never comes up as an issue for him to blame an issue on.

(You can get just the Cyberpower pigtail online if you want, or you can buy a used power brick off of eBay and it should come with the pigtail. Either way will average between $15-$20)

In this photo, you will see a cheap power inverter I bought off of Amazon for about $15. I tried to power the entire unit off of this using 12V of PoE power.

This was not entirely successful; the power brick has a momentary inrush current that exceeds what the PoE splitter is capable of handling. I believe it is associated with the filter caps that are inside the power brick charging up from a resting state. If I quickly unplugged the power brick from outside power and plugged it into the inverter, it would work just fine. I have not observed this behavior of the ONT itself; the ONT can be powered up from a cold discharged state without over-currenting the PoE splitter or PoE port on the switch.

Cheddar’s is a restaurant that I hardly go to (Other than today, I’ve been once last year with a group, and several times about 13-14 years ago). But there is one close to where I live and I fondly remember their Monte Cristo sandwiches and thought it would be a great place to try again.

I sign up and I get sent some coupons for free Chips and Queso, and a promise to send me a birthday coupon for a free desert when the time comes. Great.

I try to use these coupons but I just get told “Sorry, but this coupon has expired or not valid” (the code is not expired and the coupon is valid, unless they’re sending out invalid coupons).

I message their customer relations people who are not very helpful. They just say “Oh try using the website instead of the app”. The website gives the same error as the app.

But they tell me something curious: “The code is valid when I [the CSR] use it”, meaning they went through the process of creating an order (but presumably stopped short of actually making the order) and the code was accepted.

I try to create new accounts and get the same thing. Apparently my shit-list status carries over to new accounts (implying either an IP-address level ban on the server, or something in the app that tells them I’m the same guy when I create a new account without first uninstalling the app completely).

So here’s what I did to finally narrow this down and work around the issue:

• Uninstall the app and delete all of its data from my phone and then reinstall it. (My understanding is that apps can’t uniquely identify your device unless they request permission and you allow them to)
• Turn off WiFi (so that my connection just appears to be that of a regular NAT’d cell phone data user)
• Temporarily disable the IPSEC tunnel on my phone (one of the things I suspect might have triggered the shitlist flag)

Then I take these measures:

• Start the app but don’t give it any data that might suggest it is me. This means:
• No granular location data (iOS lets you give it imprecise location data, i.e. it might tell the app you’re a mile or two from your actual location)
• For the order, I gave it the phone number of someone (who agreed to it) with a real cell phone that isn’t tied to me and was never used for a Cheddar’s account. I thought about using a Google Voice number but thought better of it (in case they’re screening for “VoIP”-type services being used)
• I don’t actually sign into an account and just check out as a guest.

Guess what? The “invalid” or “expired” coupon suddenly works!

What gives, Cheddar’s? I can’t be accused of abusing something I haven’t even had a chance to use yet. I haven’t been kicked out of a Cheddar’s before either.

Here are some thoughts on what MIGHT have happened:

• Some apps / websites use “VPN detection” which looks at the MSS of the path between you and them. If your MSS is deemed “suspicious”, you might get flagged as a VPN and forbidden from using their services on reason you’re hiding something from them / a probable cyber-criminal / whatever. I’ve encountered a small number of websites that do this. In my case, I use an IPSEC tunnel on all my personal devices that goes to an IPSEC gateway at my own house to protect my traffic and to grant me access to self-hosted resources that I don’t open up to the Internet. Nothing nefarious about that.
  
• I’m on the mailing list of other Darden Restaurants. In particular Olive Garden sent me two birthday coupons the last time I had a birthday. I don’t know why, but I was happy to oblige their generosity. Did I get on the “shit list” for something that was entirely their fault (i.e. enjoying two birthday cake slices when maybe they only intended for me to have one?).

So my conclusion and complaint (to app developers) is this – it’s fine to want to protect your services from abuse. It’s not fine to treat paying customers as would-be criminals and then give them no clues why they’re trapped in this mess or how to resolve it.

If you think something suspicious is going on, tell them to contact support (don’t give vague errors that don’t really hint at the problem) and empower your CSRs to resolve it. Cheddar’s CSRs had no clue what was going on and didn’t know what to do other than suggest I try ordering through the website instead.

What did I actually do wrong to deserve this? I have no idea and nobody is able to tell me! I’m kinda not sure I want to continue eating here now.

This is a quick gripe about https://shinobi.video/.

tl;dr looking to install a home surveillance system using PoE cameras, prefer to use a software package instead of buying a NVR appliance. Tried installing “Shinobi” but became frustrated at poor documentation and an install script that is careless with your system.

Shinobi came recommended by a Redditor on a post.

I *tried* to install it but found it difficult, opaque, and a bit user-hostile.

I’m sure I will get flamed for my opinions but oh well I’m going to express my opinions:

• The website discourages container-based installations, for reasons not clearly stated. A container-first approach would have avoided the problem I had in the latter half of this list.
• The container installer, like the bare metal installer, is a script rather than a set of instructions. You shouldn’t trust install scripts and developers should not use them, or if they do, the install script should very clearly explain what step(s) are about to be performed.
• The container installer only recognizes docker, it does not recognize podman. Podman is a mostly-compatible fork of docker created by Red Hat and is the preferred container engine on RHEL-based distros, and is an available option in Ubuntu. (Note that Docker doesn’t even publish Red Hat packages anymore -they’ve given up)
• You can alias docker to podman, but the lack of awareness on the part of the dev seems to suggest they haven’t done testing on Red Hat distros like RHEL or Fedora.
• podman-compose isn’t (AFAIK) 100% compatible with docker-compose commands. It might work, I don’t know, but I don’t use *-compose and am not interested in it.
• They provide some docker-run commands but documentation is lacking.
• They want you to download a shell script and run the shell script to perform the installation. The actual installation steps are buried within sub-scripts that get downloaded from a git repo. Sketchy.
• The installer gives you several choices, i.e. a “touchless” installation or an “advanced” installation.
• The touchless installation is anything but. It performs many apt-gets without warning or prompting the user what is happening. In my case, it tried to download mariadb. However I am already running mysql which conflicts with mariadb.
• The apt-get commands in their scripts will not stop the installation when the conflict is detected, instead they added a force flag which uninstalls mysql and removes your existing data with no warning.

Look, I get that it is a community project. I’m sympathetic to the fact that it’s all free labor and I accept responsibility for running scripts without more careful inspection.

I got frustrated with the (lack of) good podman support and tried installing the “bare metal” version and got burned by it.

YMMV and you might have a great experience with it as opposed to me.

This is probably going to be irrelevant when Ubuntu 24.04 goes out next month, but as a quick reference for anyone who really wants this there is a handy build script out there on github.

It can also install other versions of gcc.

https://github.com/slyfox1186/script-repo?tab=readme-ov-file#install-gcc-versions-latest

certbot renewed my SSL certificates and automatically loaded them into strongswan. I am currently unable to establish an IPSEC tunnel from my iOS devices to my home VPN server with these new certificates.

Unfortunately in typical Apple tradition, there are no readily accessible logs to understand why the underlying IKE daemons in iOS are unhappy. Strongswan has no problems with the new certificates.

I examined the certificates (working vs. non-working) and see that Let’s Encrypt is now using an EC instead of RSA to generate the keypair.

Apple’s documentation states they support ECDSA certificates for IKEv2 authentication.

< Public Key Algorithm: rsaEncryption
< Public-Key: (2048 bit)
>             Public Key Algorithm: id-ecPublicKey
>                 Public-Key: (256 bit)
>                 ASN1 OID: prime256v1
>                 NIST CURVE: P-256

<                 Digital Signature, Key Encipherment
>                 Digital Signature

I found a similar reported issue here from last year: https://apple.stackexchange.com/questions/412089/ios-native-ikev2-client-and-ecdsa-server-certificates

Only other thing that is different is the Key Usage field does not include “Key Encipherment” as an allowed usage.

Changing Let’s Encrypt to force the generation of an RSA certificate resolves the issue.

I run an ADS-B monitoring station using RTL-SDR USB dongles. Normally people do this with Raspberry Pis but I have a perfectly good AMD64 box that runs 24/7.

These are the new .deb packages for this version:

http://repo.feed.flightradar24.com/linux_x86_binaries/fr24feed_1.0.34-0_i386.deb x86

http://repo.feed.flightradar24.com/linux_x86_64_binaries/fr24feed_1.0.34-0_amd64.deb amd64

I run these in a podman (docker) container. fr24feed failed to load because CAP_SYS_NICE was not enabled on the container.

You can add the capability to the container or remove the capability from the /usr/bin/fr24feed binary using the instructions here: https://gitlab.alpinelinux.org/alpine/aports/-/issues/11992. Doing either of those options will allow fr24feed to start up.

Part of the reason I’ve started this blog is so I can write down my tech notes with important details instead of forgetting the details and having to spend a couple of hours figuring things out again when I dust off a project I put on the backburner months ago.

I apologize if you came here looking for a complete guide to net booting a Pi; I only intended to write this blog to fill in a missing detail that bugs me. Here’s a couple of places that might serve as good starting points if that’s what you need:

https://tp4348.medium.com/netboot-raspberry-pi-using-ubuntu-20-04-os-cb3973ff65b0

https://xunnanxu.github.io/2020/11/28/PXE-Boot-Diskless-Raspberry-Pi-4-With-Ubuntu-Ubiquiti-and-Synology-1-DHCP-Setup/

Anyway, to the point: I got PXE boot to work. Everything boots up, I can ssh to the Pi, etc. But as someone who has deployed NFS to clients before, I noticed root is mounted as NFSv3:

# mount | grep nfs
192.168.29.2:/pxe/storepi on / type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,port=2049,timeo=600,retrans=10,sec=sys,local_lock=all,addr=192.168.29.2)

Is this really a big deal? Not really. But NFSv4 has a lot of improvements over NFSv3, including features that make the protocol more efficient. You probably won’t notice it unless you’re benchmarking. But I think it’s important to be exposed to the latest technologies so I can be familiar and apply what I’ve learned elsewhere. So thus, I *demand* that this be mounted as NFS v4.

Why is this happening? Linux will normally mount things with the highest possible NFS version that both client and server support. The client and sever both run Ubuntu 22.04 and should happily allow a NFSv4.2 mount.

The cause is found in the initramfs environment. The initramfs contains a very small working system, i.e. a few utilities that are just enough to prepare and boot into the *real* system. This includes filesystem mounting tools.

The “nfsmount” utility that ships in Debian (and is sourced by Ubuntu) does not support NFSv4. There are some ugly hacks and attempts to make it work, if you care to read the comments.

https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/1954716

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=409272

I found an easier solution: copy the /sbin/mount.nfs utility into the initramfs and call that instead of the weak-sauce “nfsmount” utility that comes by default.

What are the downsides to doing this, and why doesn’t Debian / Ubuntu just do that? Probably because the utilities that are in initramfs are intended to be *very small*. They do not even use glibc, they use klibc which is intended to produce tiny binaries.

Is there an actual downside to having a larger-than-normal initramfs? In this scenario I don’t see it but I’m guessing Debian and Ubuntu don’t want to do that due to the bloating of the initramfs.

Here are the steps:

1. Do the necessary steps to get TFTP / NFS booting of your Ubuntu / Raspbian system working. Make sure it actually works and boots successfully before proceeding further.
2. Create a “/usr/share/initramfs-tools/hooks/nfs” file with the contents shown in the code blob below. Make sure it is executable.
3. Edit the existing file “/usr/share/initramfs-tools/scripts/nfs”. Refer to the code blob below.
   
   In the function “nfs_mount_root_impl()”, comment out the “nfsmount” command and add the “mount.nfs” command that I have shown. Those are the only changes you need to make here.
   
   Note that I snipped out all but the last few lines of the function. Don’t actually snip out all of this in your file, I did that just to keep the code blob tidy. You still need all of that for this to work correctly.
4. Regenerate your initramfs using this command:
   update-initramfs -c -k all
5. Reboot. If you did this correctly, the system should come up and you should see a similar output from the “mount” command as per the below code blob showing an nfs vers=4.x mount.

root@storepi:/usr/share/initramfs-tools/hooks# cat nfs
#!/bin/sh
set -e
PREREQ=""
prereqs () {
        echo "${PREREQ}"
}  
case "${1}" in
        prereqs)
                prereqs
                exit 0
                ;;
esac

. /usr/share/initramfs-tools/hook-functions

copy_exec /sbin/mount.nfs /sbin
exit 0

from /usr/share/initramfs-tools/scripts/nfs
# parse nfs bootargs and mount nfs
nfs_mount_root_impl()
{       
        .. snipped ..
        # shellcheck disable=SC2086
        mount.nfs ${NFSROOT} ${rootmnt} -o nolock ${roflag} ${NFSOPTS}
        #nfsmount -o nolock ${roflag} ${NFSOPTS} "${NFSROOT}" "${rootmnt?}"
}

192.168.29.2:/pxe/storepi on / type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=10,sec=sys,clientaddr=192.168.29.16,local_lock=none,addr=192.168.29.2)

My home server runs Ubuntu 22.04. I do a lot of RHEL development at work. Naturally I would like to have a more diverse environment at home.

If you use Red Hat’s UBI (universal base image), you get free access to a reasonable subset of what is available from a RHEL subscription.

If the UBI subset fits your needs, that’s great. However, in the process of setting up a RHEL 9 container to host this very WordPress blog, I found some packages that WordPress suggested I install were missing. I noticed when I was trying to install the PHP ImageMagick plugin from EPEL that it had a dependency on graphviz. graphviz is available in RHEL but not available in the UBI subset. Dang.

I have a free RHEL developer subscription, however, actually subscribing a UBI image to a RHEL subscription is somewhat tricky if you aren’t running the container on a RHEL host.

If you run your container on a RHEL host, RHEL’s clone of docker, podman, will (to my understanding) bind-mount the host’s subscription certificates into the container, allowing the container to inherit the host’s subscription without having to register the container as a separate instance and consuming a subscription.

To make this work on Ubuntu, one cannot simply just do a “subscription-manager register”:

subscription-manager is disabled when running inside a container. Please refer to your host system for subscription management.

But my host system is Ubuntu!

Red Hat seems to prefer stability over compatibility so they prefer to not support containers running on non-RHEL container runtimes, hence they make it difficult to put a RHEL subscription into a container running on a non-RHEL runtime host.

I found this article: https://access.redhat.com/discussions/5889431

Apparently the simplest thing to do is run this sed one-liner from a user in that discussion to modify the Python code that checks if the container is running on a RHEL host to always convince it that it’s not in a container:

sed -i 's/\(def in_container():\)/\1\n    return False/g' /usr/lib64/python*/*-packages/rhsm/config.py

Note that this is fine if you’re putting this kind of container into a test / dev / lab environment, but not a good idea to make it “production” because it’s not supported and support will turn you away if you try to open a case on such a container.

However it is working fine for my home lab use case and I am happy to have it as a way to “broaden my horizons” so to speak.

I spent a lot of time running old OSes, mainly for personal amusement and not for doing any practical work. I guess the lack of practicality is why I’m running into bugs / issues with qemu and older OSes.

Windows NT 4, released in 1996, does in fact support SMP. However, qemu with KVM acceleration will crash if you install the Multiprocessor HAL in Windows NT 4.

Stack trace of thread 2072708:
#0  0x00007f2e8a90ba7c __pthread_kill_implementation (libc.so.6 + 0x96a7c)
#1  0x00007f2e8a8b7476 __GI_raise (libc.so.6 + 0x42476)
#2  0x00007f2e8a89d7f3 __GI_abort (libc.so.6 + 0x287f3)
#3  0x000055cbf99545a2 do_patch_instruction (qemu-system-i386 + 0x2f85a2)
#4  0x000055cbf995b1d4 process_queued_cpu_work (qemu-system-i386 + 0x2ff1d4)
#5  0x000055cbf9d762f8 kvm_vcpu_thread_fn (qemu-system-i386 + 0x71a2f8)
#6  0x000055cbf9ef4560 qemu_thread_start (qemu-system-i386 + 0x898560)
#7  0x00007f2e8a909b43 start_thread (libc.so.6 + 0x94b43)
#8  0x00007f2e8a99ba00 __clone3 (libc.so.6 + 0x126a00)

do_patch_instruction is located in kvmvapic.c. It contains a switch block that evaluates a CPU instruction given to it, based on opcode, and if the opcode of the instruction does not match anything in the switch block, it will fall through to the default case and call libc’s abort().

Whatever NT is doing, it goes through this code and feeds it an instruction it doesn’t understand. The qemu devs had their reasons for calling abort() I suppose.

Comment out, recompile, and bam Windows NT will boot correctly now!

402 static void do_patch_instruction(CPUState *cs, run_on_cpu_data data)
403 {   
404     X86CPU *x86_cpu = X86_CPU(cs);
405     PatchInfo *info = (PatchInfo *) data.host_ptr;
406     VAPICHandlers *handlers = info->handler;
407     target_ulong ip = info->ip;
408     uint8_t opcode[2];
409     uint32_t imm32 = 0;
410     
411     cpu_memory_rw_debug(cs, ip, opcode, sizeof(opcode), 0);
412     
413     switch (opcode[0]) {
414     case 0x89: /* mov r32 to r/m32 */
415         patch_byte(x86_cpu, ip, 0x50 + modrm_reg(opcode[1]));  /* push reg */
416         patch_call(x86_cpu, ip + 1, handlers->set_tpr);
417         break; 
418     case 0x8b: /* mov r/m32 to r32 */
419         patch_byte(x86_cpu, ip, 0x90);
420         patch_call(x86_cpu, ip + 1, handlers->get_tpr[modrm_reg(opcode[1])]);
421         break; 
422     case 0xa1: /* mov abs to eax */
423         patch_call(x86_cpu, ip, handlers->get_tpr[0]);
424         break; 
425     case 0xa3: /* mov eax to abs */
426         patch_call(x86_cpu, ip, handlers->set_tpr_eax);
427         break; 
428     case 0xc7: /* mov imm32, r/m32 (c7/0) */
429         patch_byte(x86_cpu, ip, 0x68);  /* push imm32 */
430         cpu_memory_rw_debug(cs, ip + 6, (void *)&imm32, sizeof(imm32), 0);
431         cpu_memory_rw_debug(cs, ip + 1, (void *)&imm32, sizeof(imm32), 1);
432         patch_call(x86_cpu, ip + 5, handlers->set_tpr);
433         break; 
434     case 0xff: /* push r/m32 */ 
435         patch_byte(x86_cpu, ip, 0x50); /* push eax */
436         patch_call(x86_cpu, ip + 1, handlers->get_tpr_stack);
437         break;
438     default:
439         //abort();
440     }
441     
442     g_free(info);
443 }

There is another workaround for this: you can not use KVM and just use qemu’s TCG acceleration. Which is likely slower. Otherwise you have to limit yourself to one CPU if you want KVM.

One last problem with Windows NT 4.0: the multiprocessor HAL, unlike the uniprocessor HAL, does not sent a HLT instruction to idle CPUs. Meaning, like back in the Windows 95 days, your CPU will burn cycles doing nothing.

This person discusses the issue and resolved it by disassembling the hal.dll and adding the HLT instruction. He offers a replacement version of the hal.dll, but I did not have luck (the system boots and sends the HLT as expected, but it’s extremely sluggish).

Maybe I’ll investigate further. Or maybe not, it’s not that important.