I have battled this problem a couple times and so I am documenting the issue and solution here.
Here is the error message from make modules_install

ln: target '+/source': No such file or directory

The short solution is to watch out for stray whitespace in the Makefile

This happens when I attempt to modify the Makefile in order to revision control the Kernel. I I suspect that a build I am about to make will prevent the machine I am working on from booting, I want to keep an older build functional in order to restore the machine.

Here is the diff for my Makefile:

diff --git a/Makefile b/Makefile
index 2fdd8b40b7e0..cb747c16e33c 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 VERSION = 6
 PATCHLEVEL = 5
-SUBLEVEL = 0
+SUBLEVEL = 5
 EXTRAVERSION =
 NAME = Hurr durr I'ma ninja sloth

When the run fails, the diff will show a stray whitespace character around the SUBLEVEL value. This value is used to generate the string the becomes part of the path where the install happens. At the top level, Kernels get installed into here:

# ls -la /usr/lib/modules/
total 20
drwxr-xr-x.  8 root root  122 Sep 25 08:15 .
dr-xr-xr-x. 34 root root 4096 Sep 18 09:51 ..
drwxr-xr-x.  8 root root 4096 May 15 19:06 5.17.5-300.fc36.aarch64
drwxr-xr-x.  8 root root 4096 Sep 18 09:53 6.2.15-100.fc36.aarch64
drwxr-xr-x.  3 root root 4096 Sep 22 10:26 6.5.0+
drwxr-xr-x.  2 root root    6 Sep 23 07:42 6.5.1
drwxr-xr-x.  2 root root    6 Sep 23 07:57 6.5.5
drwxr-xr-x.  3 root root 4096 Sep 25 08:18 6.5.5+

The source directory is a symlink underneath it:

# ls -la /usr/lib/modules/6.5.5+/source
lrwxrwxrwx. 1 root root 11 Sep 25 08:15 /usr/lib/modules/6.5.5+/source -> /root/linux

Thus by having an extra space anywhere in there, the path generation gets messed up.

In the software development world, we call it technical debt.


In the Army it was “Half-assed, full-blast. Don’t know where we are going but we should have been there yesterday.”

And the solution was told to be by a guy going through officer basic with me…after a long career as an NCO in Army Special Forces.


“Make Haste Slowly.”


It is ok to “just make it work.” But have a strict enough code review process that is happy to kick back semi-functional code to get it production quality.

Think of it like an English Essay: it is ok to show your teacher a rough draft, but expect lots of Red Ink and rewriting on it.


Unit Test Everything. Automated testing will catch when your change code that breaks other code. Visual and manual testing does not count. It has to be automated or it is not sufficient. Not writing unit tests is heavy tech debt.

To find out what line a particular stack trace entry points to, use the script ./scripts/faddr2line for example If I have the line __get_vm_area_node+0x17c/0x1a8 I can run

./scripts/faddr2line vmlinux.o __get_vm_area_node+0x17c/0x1a8
__get_vm_area_node+0x17c/0x1a8:
__get_vm_area_node at /root/linux/mm/vmalloc.c:2579 (discriminator 1)

Sometimes you don’t get to use a debugger. When do bare metal development, often it is faster to get to the root of a problem by throwing in trace statements, and seeing what path is taken through the code.

There are two main techniques I have been using to do this. The first is to print out the spot in the code using built in macros that tell the file, the name of the function, and the line number. That looks like this:

pr_info("%s %s %d", __FILE__, __func__,  __LINE);

I know it looks a little weird having some upper and some lower case in there, but that is what works.

However, Linux makes heavy use of function pointers, and you cannot use tags to jump to a function whose name you do not know. To print out the source of a function from a pointer, you can use the print formatting macros specific to the Linux Kernel. For example: I can use

printk("%ps", pmu->event_init);

In my case, that prints out:

arm_cspmu_event_init [arm_cspmu_module]

Which I could then jump to using the :tag command in vim.

According to the puzzles coder, any puzzle can be solved in 3 clicks (or less). That means that following my algorithm is likely to be significantly less efficient than an optimum solve.

I am not sure I buy it…yet.

Lets see if we can optimize a solve path for a puzzle.

To simplify the notation, I am going to label the boxes using Hexidecimal notation.

• The top left box is 0,
• the one next to it is 1.
• the one at the end of the row is 3
• the first one in the second row is 4
• The last one is F

A solve can thus be listed as a stream of hexidecimal characters. For example, this sequence C 8 C D C E F B F 8 9 D C can solve the puzzle below.

That is 13 clicks, 10 more than the proposed optimum.

However, If I then go to swap cells 4 and E, I end up clicking C 3 times on every other click. Something tells me this is going to be redundant. The same is true for the bottom right: I end up clicking F Twice in alteration. And I end up back on D C.

I note that this sequence now also solves the puzzle:

8 D E B 8 9 D

7 clicks. Over a 40% improvement.

What about that D? What if we skip that? Yep.

8 E B 8 9 Solves it. That is less than half the number of clicks we started with. Can we go further? What if we drop those 8s?

Yep. E B 9 Solves it as well.

It seems like the pattern is to remove any double clicks. Lets try another puzzle and see.

Puzzle 65535 is all black. Using my algorithm there are no repeated clicks. So that does us no good. But it implies that there must be a way to transition from one solve pattern to another.

The algorithmic solution to this puzzle is

0 2 3 4 0 3 2 6 7

If we drop the repeated 0 2 and 3 we get

4 6 7

Which also is a solution.

The blank puzzle (puzzle 0) can be solved with

0 2 5 4 2 3 7 6 8 9 D C A B F E 0 3 F C

What if we drop all of the duplicates?

547689DABE

It does not work…but it is close…if we add a further 2 3 at the very end IT does. Curious.

The following pattern also solves it: Click the three points around the corner, but not the corner itself. In ANY ORDER.

12456789ABDE

Does that mean that order is meaningless? Lets go back to our original puzzle # 4079 :

EB9 E9B 9BE 9EB Be9 B9E

Yes. They all solve it.

So it appears that the 3 clicks to solve might have some merit. We need a way to convert from non-overlapping solutions to solution that overlap with redundancy. This means we might well got from an algorithmic solution to a less efficient solution, and then from there to a more efficient solution.

Edit: Rereading what he said, I see now that it was “any daily or random” puzzle. Which are, I am guessing, generated using the algorithm of “start with the solution, and randomly flip three dots.”

My Friend Evan Barr coded up a puzzle he had read about in a Mensa magazine. I suggest you click around on it a bit before reading on.

Techniques to Solve

Here is how I went about solving it, not just for one incarnation, but for any.

If we number the Grid like a Sudoku puzzle, we have box (1,1) through (4,4).

Swapping the corners

Start by setting all four corners to black by clicking them directly. This will Mess up other dots, but we will fix them later.

Swapping interior dots

There are a series of click patterns that have desired results . The first of these is for switching the state of any of the four interior dots.

If you click each of the four dots connected to a corner in either a clockwise or a counterclockwise fashion, you can swap the state of the dot in center diagonally connected.

Stated another way, clicks [(1,1),(1,2),(2,2),(2,1)] will switch (3,3). Every other dot on the board remains in the initial state.

Swapping internal side blocks.

Once you know the interior technique, you do not need to worry about maintaining the state of the interior blocks. Thus, you can use a more destructive technique to swap any of the edge blocks not in a corner. For example, to swap block (1,3), you can click on (1,2) and then block (1,1). This Will also swap block (2,3) but you can reverse that with the interior dots technique.

Chaining techniques

Once you have these set of techniques, you can chain them to solve the whole puzzle.

• Swap the corner blocks so they are all the same.
• Swapping just the edge internal blocks to get the external and internal dots in opposition.
• Follow by using Swapping interior dots to set all of the interior dots to the same color.

Additional Techniques

You won’t need these techniques but they are neat.

Swap The whole grid

It should be trivial to see that clicking in any of the corners will swap all of the blocks in the four blocks connected to the corner. Clicking all four corner blocks swaps the entire state of the grid.

Swapping just the corner blocks.

To swap the states of just the four corner blocks, click the four internal blocks one at a time in either a clockwise or counterclockwise patterns. [(2,2), (2,3), (3,3), (2,2)]

Swapping just the edge internal blocks.

To swap the blocks on the edges that are not corner blocks, click each of them in sequence in either a clockwise or counterclockwise pattern. e.g. [(1,2),(1,3),(2,3),(3,4),(4,3),(4,2),(3,1),(2,)]

Note that you need to have a .config file that will be included in the build. It will also use the Version as specified in your Makefile. Then run

make rpm-pkg

Which will use the RPM build infra set up for your user to put the rpm in $HOME/rpmbuild/

I need to write a driver for a device that does not exist yet. So, I am going to use the Linux kernel tooling fro ACPI to create the illusion that the device exists. Here is how.

Devices on an ACPI enabled Linux machine typically exist in a able called the DSDT. This has a real name, but I just thing of it as “Different Stuff Different Table”. However, you can also have a machine specific table that you create at boot time and put entries in there. This is called the SSDT. Again, it has a real name, but I just think of it as “Same Device Different Table” Here is my simple SSDT definition in the ACPI domain specific language.

 /*
 * Template for [SSDT] ACPI Table (AML byte code table)
 */
DefinitionBlock("","SSDT", 2, "Ampere", "_SSDT_01", 0x00000001)
{

    Scope(\_SB)
    {
       Device(MCTP)
       {
           Name (_HID, "MCTPA1B2")

           Name (_STA, 0x0F)
       }
       Device(PLDM)
       {
           Name (_HID, "PLDMA1B2")

           Name (_STA, 0x0F)
       }
       Device(PSTL)
       {
           Name (_HID, "POSTA1B2")

           Name (_STA, 0x0F)
       }
     }

}

Now, this actually has three devices in it, and I was doing some unit testing setup with dependent devices so I though I might use the other ones. I left them in as an example to show how multiple devices look in an SSDT.

To convert this to the binary format, I use iasl, a utility in the acpica-tools RPM. Yes, I still do RPM, although all of this works on Debian based systems as well.

Here is my script to load it into the Linux kernel. It uses a module for the ACPI configuration filesystem.

#! /bin/sh 
modprobe acpi_configfs
mkdir /sys/kernel/config/acpi/table/ssdd
cat ~/acpi/bak/mctpdev.aml > /sys/kernel/config/acpi/table/ssdd/aml

To confirm that the file exists, you can use the acpidump command. If you run it with the -b flag you get all the tables in binary format.

[root@hackery tmp]# mkdir acpi
[root@hackery tmp]# cd acpi/
[root@hackery acpi]# acpidump -b

The iasl command will then decompile the table and you can view the contents. I’ll leave you with the full content.

[root@hackery acpi]# iasl ssdt.dat 

Intel ACPI Component Architecture
ASL+ Optimizing Compiler/Disassembler version 20220331
Copyright (c) 2000 - 2022 Intel Corporation

File appears to be binary: found 31 non-ASCII characters, disassembling
Binary file appears to be a valid ACPI table, disassembling
Input file ssdt.dat, Length 0x83 (131) bytes
ACPI: SSDT 0x0000000000000000 000083 (v02 Ampere _SSDT_01 00000001 INTL 20220331)
Pass 1 parse of [SSDT]
Pass 2 parse of [SSDT]
Parsing Deferred Opcodes (Methods/Buffers/Packages/Regions)

Parsing completed
Disassembly completed
ASL Output:    ssdt.dsl - 1141 bytes
[root@hackery acpi]# cat ssdt.dsl 
/*
 * Intel ACPI Component Architecture
 * AML/ASL+ Disassembler version 20220331 (64-bit version)
 * Copyright (c) 2000 - 2022 Intel Corporation
 * 
 * Disassembling to symbolic ASL+ operators
 *
 * Disassembly of ssdt.dat, Sat Jul  8 04:42:31 2023
 *
 * Original Table Header:
 *     Signature        "SSDT"
 *     Length           0x00000083 (131)
 *     Revision         0x02
 *     Checksum         0xCE
 *     OEM ID           "Ampere"
 *     OEM Table ID     "_SSDT_01"
 *     OEM Revision     0x00000001 (1)
 *     Compiler ID      "INTL"
 *     Compiler Version 0x20220331 (539099953)
 */
DefinitionBlock ("", "SSDT", 2, "Ampere", "_SSDT_01", 0x00000001)
{
    Scope (\_SB)
    {
        Device (MCTP)
        {
            Name (_HID, "MCTPA1B2")  // _HID: Hardware ID
            Name (_STA, 0x0F)  // _STA: Status
        }

        Device (PLDM)
        {
            Name (_HID, "PLDMA1B2")  // _HID: Hardware ID
            Name (_STA, 0x0F)  // _STA: Status
        }

        Device (PSTL)
        {
            Name (_HID, "POSTA1B2")  // _HID: Hardware ID
            Name (_STA, 0x0F)  // _STA: Status
        }
    }
}

When talking about Rocky Linux or other distros based on RHEL,
I don’t think that it is a good idea to paint any one person as the bad guy or that they are the bad guy or operating in bad faith. I do not think this is the case with any of the people behind Rocky.

I was in sales at Red Hat during this time, and I winced when I heard the announcement. While Stream was and is a great step forward, closing off the Open But unsupported CentOS option was bad for Red Hat’s business. Instead of walking into an organization built around Yum and RPM, and the tools that managed them, we’d end up waling into organizations built around apt and .deb.

As A Linux guy, I don’t care. As Red Hat employee, I did.

Rocky provides a need for people that need to have their stuff behave like RHEL but are doing something that means that Red Hat support is an unnecessary expense. For example, when I worked at Penguin Computing, we needed our stuff to act like RHEL, as it was going to be loaded on machines that could be rebooted to RHEL, but we were doing a custom Kernel module (BProc) that was not going to be accepted upstream. Loading it into a RHEL system would taint the Kernel.

We were PXE booting a large array of very NON-RHEL systems from this one. We could not afford the expense of RHEL suport for 200+ Nodes that were only installed with our own Code. CentOS was much easier for us than Rolling our own RPMS.

Did we benefit from Red Hat effort? Yes. Did we contribute back to the community? As much as my 9 person development team could. Aside from the base OS we also had significant effort put into scientific libraries and Message Passing subsystems. We were stretched thin.

Rocky is Good for Red Hat. If I was back and sales and I knew that a customer team was based on Rocky, I would know how to get them running on RHEL if the opportunity arose. I would know that there stuff would run. That is a huge benefit to the sales process.

If no one is calling Red Hat data centers for support or filing customer support tickets for Rocky, it costs Red Hat nothing. If people find bugs on Rocky and file them against RHEL in RH bugzilla, it is good for Red Hat.

Stream is also a good effort, but it does not fulfill the need that Rocky does.

xset led on

As Jazz students, we are often exhorted to take a lesson and practice it in all 12 Keys. How many ways can we take a pattern through all 12 keys? Lets do some math.

Bottom Line Up Front: The easiest options are chromatic or cycle of fifths. Here’s how I justify that statement.

Lets map each note to a number, using an analog clock face positions of the numbers for each note, starting with A = 1.

The number 12 is very useful as a collection of things, because it can be evenly broken up in many ways:

• 2 Groups of 6
• 3 groups of 4
• 6 Groups of 2
• 4 groups of 3

However, this means that if we try to use one of those numbers as the basis for skipping notes, we will not hit every key. For example, take the “two groups of 6” option. That is what you get if you use the Whole tone scale. If You start at 1 (A) you will get 1,3,5,7,9,11 and then back to 1. To hit every key you would then have to do a half step or a step and a half to get the other 67 notes. Four groups of 3 is the augmented triads. 3 groups of 4 is the diminshed triads.

Add in the fact that 8 and 9 are multiples of 3 and 4 and that leaves use with a couple options:

• Plus or Minus 1
• Plus or Minus 5
• Plus or Minus 7

These map to the most common ways we are told to practice in all 12 keys: Chromatically or the cycle of fifths. It should be pretty obvious how plus 1 means a half step up; Start on A, then on A# etc. This is more common than Minus 1, or a half step down, but the effect is fairly similar.

If we start with the fact that 5 plus 7 = 12 it should not then be a surprise that +7 gets the same pattern as minus 5. 1 + 7 = 8. Going backwards around the clock is easiest if we add 12 before cross the 12 boundary. So 1 + 12 – 5 = 13 – 5 = 8.

So plus five is an interval of a Fourth ( A to D) as is minus 7.

Plus 7 is an interval of a fifth (A to E) as is minus 5.

So if we want an unbroken pattern that is going hit every key, those are our only options…or we can make an irregular pattern that adds up to 5 or 7. For example, +3 +2, which would be a minor third followed by a whole step. It can also be larger intervals: For example

• +6 + 1,or tritone, half step.
• +7 +6: perfect 5th.
• +6 +5 tritone perfect fourth

Those work because the forth/fifth shifts between the two whole tone scales.

This Means we could also do +4 +3: major third minor third. This Is Roughly based on a major triad; The final note of one is the Root of the next.

I suspect that it was pondering along these lines that lead to the Coltrane Circle.

Never use that phrase with me again, please. It is a most insulting phrase.

It is saying “You are wrong, and you are not even worth engaging with to try and change your mind. I am always willing to listen to a strong argument. I do not agree to disagree with you; I agree to hold on to my opinions until shown a superior one. I agree to keep trying to change your mind until it is changed, or until you provide a better argument.

That phrase has been used on me too many times to dismiss my concerns. It is associated with one of the worst job interviews of my career.

Either argue your point or stay silent.

I do not agree to disagree with you in perpetuity. I disagree with you now. I am willing to put in the effort to bring you around to my view point. I am willing to listen to your point of view.

If you are not willing to engage with me, do not engage with me.

I am far more likely to say “I am not willing to pursue this argument.” Which I often will do for things that may be too painful to discuss. That is usually politics, religion or other sensitive matters like that. But that is almost always an issue of self-preservation.

This phrase has not value and you should excise it from your lexicon.

If your disagree, you better have your argument in order.

A couple months back I recorded this line on my blog as part of investigating perf:

perf record --branch-filter any,save_type,u true 

What is the interface between the perf binary and the linux Kernel when I run this? There is a system call to open a file handle. The man page says this:

int syscall(SYS_perf_event_open, struct perf_event_attr *attr,
                    pid_t pid, int cpu, int group_fd, 
                    unsigned long flags);

But how does that get called by the perf binary…the answer is trickier than I originally thought.

When debugging system calls, the primary tool I use is strace. I ran it like this:

sudo strace  perf record --branch-filter any,save_type,u true  2>&1  | less

Inside of less, I can search for the string “event_open” and I am taken to a spot in the output where the system call is made…11 times. Each one opens a file handle, and all of them are kept open after the system call. Here is the first call:

perf_event_open({
                   type=PERF_TYPE_SOFTWARE, 
                   size=0 /* PERF_ATTR_SIZE_??? */,
                   config=PERF_COUNT_SW_CPU_CLOCK,
                   sample_period=0,
                   sample_type=0,
                   read_format=0,
                   exclude_kernel=1,
                   precise_ip=0 /* arbitrary skid */, ...},
                 -1,
                  2,
                 -1,
                  PERF_FLAG_FD_CLOEXEC) = 4 

And here is the last one

perf_event_open({
                   type=PERF_TYPE_HARDWARE, 
                   size=PERF_ATTR_SIZE_VER7,
                   config=PERF_COUNT_HW_CPU_CYCLES,
                   sample_freq=4000,                  sample_type=PERF_SAMPLE_IP|PERF_SAMPLE_TID|PERF_SAMPLE_TIME|PERF_SAMPLE_ID|PERF_SAMPLE_PERIOD|PERF_SAMPLE_BRANCH_STACK,
                   read_format=PERF_FORMAT_ID, 
                   disabled=1, 
                   inherit=1, 
                   mmap=1, 
                   comm=1, 
                   freq=1, 
                   enable_on_exec=1, 
                   task=1,
                   precise_ip=3 /* must have 0 skid */,
                   sample_id_all=1,
                   exclude_guest=1,
                   mmap2=1,
                   comm_exec=1,
                   ksymbol=1,
                   bpf_event=1,
                ...},
                11127,
                7,
               -1,
                PERF_FLAG_FD_CLOEXEC) = 12 

I’ve formatted this to make the relationship of the parameters a little clearer. Most significant is that the first parameter is a large structure, which strace knows how to interpret.

If we keep looking down the output of strace, we can see another block of perf_event_open. When all of these are executed we end up with file descriptors 4-12 and 13-20 open for the remainder of the program as products of the perf_event_open system call. These are never read from, so the information must come from the kernel via another means. Again, we can see a block of related system calls, this time ioctls:

ioctl(13, PERF_EVENT_IOC_ENABLE, 0)     = 0
ioctl(14, PERF_EVENT_IOC_ENABLE, 0)     = 0
ioctl(15, PERF_EVENT_IOC_ENABLE, 0)     = 0
ioctl(16, PERF_EVENT_IOC_ENABLE, 0)     = 0
ioctl(17, PERF_EVENT_IOC_ENABLE, 0)     = 0
ioctl(18, PERF_EVENT_IOC_ENABLE, 0)     = 0
ioctl(19, PERF_EVENT_IOC_ENABLE, 0)     = 0
ioctl(20, PERF_EVENT_IOC_ENABLE, 0)     = 0

But we do a see a clone system call, which implied a child process or thread.

clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x7faf329ef910, parent_tid=0x7faf329ef910, exit_signal=0, stack=0x7faf321ef000, stack_size=0x7ffec0, tls=0x7faf329ef640} => {parent_tid=[11539]}, 88) = 11539

If we look for system calls based on this pid (11539) we later see:

poll([{fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN|POLLERR|POLLHUP}, {fd=16, events=POLLIN|POLLERR|POLLHUP}, {fd=17, events=POLLIN|POLLERR|POLLHUP}, {fd=18, events=POLLIN|POLLERR|POLLHUP}, {fd=19, events=POLLIN|POLLERR|POLLHUP}, {fd=20, events=POLLIN|POLLERR|POLLHUP}], 8, 1000 <unfinished ...>

However, the only time this pid is referenced again is to clean up, in this block of syscalls:

[pid 11539] <... poll resumed>)         = 0 (Timeout)
[pid 11539] rt_sigprocmask(SIG_BLOCK, ~[RT_1], NULL, 8) = 0
[pid 11539] madvise(0x7faf321ef000, 8368128, MADV_DONTNEED) = 0
[pid 11539] exit(0)                     = ?
[pid 11539] +++ exited with 0 +++

MCTP stands for Management Component Transport Protocol. While it is designed around a server, it is also designed as a network protocol. As such, the Linux implementation makes use of the Socket interface and the Kernel plumbing for dealing with network protocols. To support a new transport mechanism, you implement a new device driver specific to that transport mechanism that implements the struct net_device contract, which includes implementing the functions for struct net_device_ops.

Before I write a full implementation, I want to write something that only echos a packet back to the receiver. This mimics the behavior of the mctp-echo server in the user tools, and can make use of the mctp-req echo client.

A network driver moves traffic from the operating system to the network device or in the opposite direction. My proof-of-concept code is going to do half of each operation: it is going to take a packet from the operating system, munge it, and send it back to the operating system. In order to perform this operation, I hae to implement the function .ndo_start_xmit from the struct net_device_ops. Here’s my implementation:

static netdev_tx_t mctp_pcc_tx(struct sk_buff *skb, struct net_device *ndev)
{
        struct mctp_hdr * hdr;
        struct control_work_data * control_write_data;

        u8      orig_dest;
        u8      orig_src;
        
        printk(KERN_INFO "mctp_pcc_tx called\n");

        hdr = mctp_hdr(skb);
        
        printk("Version:     %x\n",hdr->ver);
        printk("Source:      %x\n",hdr->src);
        printk("Destination: %x\n",hdr->dest);
        printk("flags_seq_tag:      %x\n",hdr->flags_seq_tag);
        pkt_hex_dump(skb); 


        orig_dest = hdr->dest;
        orig_src = hdr->src;

        __mctp_cb(skb);

        hdr = mctp_hdr(skb);
        hdr->src=orig_dest;
        hdr->dest=orig_src;
        hdr->flags_seq_tag += 8;
        //This is a single static bloc instead of a kmalloc'ed block. The kmalloc
        //Was triggering a check during interrupt context and stack dumping
        control_write_data = &control_write_data_static;//kmalloc(sizeof(struct control_work_data), GFP_KERNEL);
        control_write_data->skb = skb;
        INIT_WORK(&control_write_data->work, temp_echo_handler);
        init_waitqueue_head(&control_write_data->wq);

        schedule_work(&control_write_data->work);

        return NETDEV_TX_OK;
}

A few observations.

• I am assuming that this code is supposed to consume and free the sk_buff that is handed in. In stead of doing that, I reuse it for the outward journey. I’d like to confirm that this is correct, but I have not gotten a double free message, so I think I am right.
• I swap the source and destination values., as that is where the response message needs to go. The original destination value lets me set a routing rule to ensure that the packet gets sent to the new device driver. The recvfrom call in the mctp_req program looks for a packet from this address
• There is a flag on the header that indicates that this is a response packet. Without this, the packet does not get delivered back to the the mctp_req process. That is what this code does: hdr->flags_seq_tag += 8; Should be a bitwise or setting, but this proves the concept.
• Since the start_tx call is done in interrupt context, we want to exit out as quickly as possible. The essential work that has to be done here is scheduling the work that cannot be done in interrupt context. This is done in the function linked in the control_write_data block with the skbuff data attached. I could have put more of this work into the callback function, but nothing I did here would block.

The work_queue function that sends the packet back to the kernel looks like this:

void temp_echo_handler(struct work_struct *work){
        struct control_work_data *my_data = container_of(work, \
                                 struct control_work_data, work);
        pr_info("%s\n", __func__);
        if (my_data->skb != NULL){
                pr_info("%s calling netif_rx\n", __func__);
                netif_rx(my_data->skb);
        }
 
        pr_info("%s after netif_rx \n", __func__);
}

Aside from the logging, the function calls netif_rx(my_data->skb) which sends the packet back into the kernels’ network processing code…it ends up on a queue to get delivered to the appropriate waiting socket.

To set up a test for this code, I run the following shell script:

#!/bin/sh
insmod mctp_pcc.ko 
mctp route add 9 via mctpipcc2d
mctp address add 10 dev mctpipcc2d
mctp link set mctpipcc2d  up

With this run, I can see the mctp device using the mctp tool:

# mctp link
dev lo index 1 address 0x00:00:00:00:00:00 net 1 mtu 65536 up
dev mctpipcc2d index 9 address 0x(no-addr) net 1 mtu 68 up

To avoid fooling myself, I need to ensure that the mctp-echo server is not running, or it will respond for me.

A test run looks like this:

# ./obj/mctp-req eid 9
req:  sending to (net 1, eid 9), type 1
-> sent
req:  message from (net 1, eid 9) type 1 len 1: 0x00..

I added the ->sent message from debugging, but otherwise this is the vanilla code linked above.

Getting System Tap up and running on F38 has involved chasing down a few modules.

I am running on

uname -a
Linux hackery 6.2.15-300.fc38.aarch64+debug #1 SMP PREEMPT_DYNAMIC Thu May 11 16:10:31 UTC 2023 aarch64 GNU/Linux

I had to get the kernel-debug-devel package in in order to get the files under /usr/src/kernels/6.2.15-300.fc38.aarch64+debug/. Running stap looks for a file in /lib/modules/6.2.15-300.fc38.aarch64+debug/build which is a symlink to this directory.

sudo yum install kernel-debug-devel kernel-debug-modules kernel-debuginfo kernel-debug-core  kernel-devel kernel-devel

With that installed I can run the command:

stap --dump-probe-types

I knew that things were OK once I could get the stap-prep command to to work. That involved getting the debug Kernel running and the appropriate other RPMS. As of now I am running This version of the Linux Kernel from the Fedora RPMs 6.2.15-300.fc38.aarch64+debug.

The funny thing is that I ended up not using System Tap after all this, but I wanted to record the thing I learned.

Design prototypes for a band logo

All images Copyright Adam Young.

Designed in Inkscape. Working title is “Smirking Sigma”

The font used is Beachman Script. by David Rakowski. It really seems to capture the classic feel of a fifties music venue.

Working on a different architecture from my Laptop means that I am invariably working on a remote machine. My current development can be done on an Ampere AltraMax machine, which means I have 80 processors available; quite a nice benefit when doing a Linux Kernel compile that can use all of the processors available.

Because the machine is a shared resource out of out lab, I want to make sure I can recreate my work there on another machine; this one could be reclaimed or powered down due to lab priorities. Thus, all my remote work is done in git, and I use the ssh protocol to pull changes from my work server to my laptop fairly regularly…whenever I feel I have valuable work that could be potentially lost.

I will assume that you can figure out how to create a git repo on work machine (write a comment if you really feel that would be necessary to expand) and will just show how to pull it onto my laptop.

Here are the steps:

• create a local git repo.
• add your remote server ssh login as a git remote
• git fetch from the remote server.

Here’s what it looks like:

mkdir myproj
cd myproj
git remote add eng16sys root@eng16sys:devel/myproj
git fetch eng16sys

That last command will generate output like this:

remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 13 (delta 3), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (13/13), 3.07 KiB | 785.00 KiB/s, done.
From 10.76.105.76:devel/mctp_pcc
 * [new branch]      main       -> eng16sys-r105/main

This is a two-way setup in that I can push code from my laptop to the server as well. However, since I don’t have ssh from the server back to my laptop, I cannot pull code to the server from my laptop from a command prompt on the server. This is security constraint; I don’t want to allow people that have access to my server (a shared resource) to also have access to my personal laptop.

How do you know what is inside your computer? There are a couple tools. If the hardware is on the PCI bus, from the command line you can run lspci, which will in turn enumerate the discovered devices on that bus. But what if the hardware is not on the PCI bus? And how does the Kernel discover it in the first place? For the hardware that I have to work with, the answer is that it is enumerated by the Unified Extensible Firmware Interface (UEFI) coded embedded in the device and exposed via the Advanced Configuration and Power Interface (ACPI). This world is full of four letter acronyms. Here are my notes on some of them.

What I am about to write about is based on what I’ve learned from looking at Ampere System(s)-on-a-Chip (SoC). While details change from generation to generation, this is the general pattern I’ve learned. It is not specific to ARM servers, but it is common in ARM servers, or so I’ve been told by people who have spent more time in this world than I have.

Adam, we ain’t in Keystone anymore.

We don’t tend to call them Central Processing Units (CPUs) anymore. It seems kinda silly to call them Central when there are dozens or hundreds of them on a single SoC. We’ll call them cores. When a core needs to talk to another component on the SoC it uses some form of interconnect. The one that I have used is called Platform Communication Channel (PCC).

There are lots of channels, on a board for communication between many components. There are two items of note in a PCC; the mailbox and the doorbell. I love that these two terms are NOT ACRONYMS. They are, of course, analogies, and good ones, if not perfect. A mailbox is a shared memory location (offset and length in physical memory). A Doorbell is a register. The idea is that you put your message in the mailbox, and then push the doorbell.

The analogy breaks down when you realize that the doorbell is not just a bit, but rather a full number that can carry additional information about the package that was left in the mailbox. And, because this is a multi user operating system, there is a locking and synchronization process involved.

The Channel is a generic communication mechanism. On an AltraMax system it is used by (at least) the xgene_hwmon Linux kernel module and the cppc_cpufrew kernel module. hwmon is for temperature and power monitoring. The PPC in CPPC is not the same PPC as above. It stands for Collaborative Processor Performance Control.

ACPI information is exposed in what the spec calls tables. For the most part, these are normalized tuples, but there is some nesting. Two tables of interest for communication between parts of an Ampere System on a Chip (SoC) are the Differentiated System Description Table (DSTD) and the Platform Communication Channel Table (PCCT.) I wanted to take a closer look at what is in these tables. To do so, I used acpidump with the -b switch to extract the tables and the iasl command to convert these two tables to their human readable source code forms.

The PCCT table has a short header, and then a repeated series of entries like this one:

[032h 0050   4]           Platform Interrupt : 00000038
[036h 0054   1]        Flags (Decoded Below) : 00
                                    Polarity : 0
                                        Mode : 0
[037h 0055   1]                     Reserved : 00
[038h 0056   8]                 Base Address : 00000000F0C10000
[040h 0064   8]               Address Length : 0000000000004000

[048h 0072  12]            Doorbell Register : [Generic Address Structure]
[048h 0072   1]                     Space ID : 00 [SystemMemory]
[049h 0073   1]                    Bit Width : 20
[04Ah 0074   1]                   Bit Offset : 00
[04Bh 0075   1]         Encoded Access Width : 03 [DWord Access:32]
[04Ch 0076   8]                      Address : 0000100001540010

[054h 0084   8]                Preserve Mask : 0000000000000000
[05Ch 0092   8]                   Write Mask : 0000000053000040
[064h 0100   4]              Command Latency : 000003E8
[068h 0104   4]          Maximum Access Rate : 00000000
[06Ch 0108   2]      Minimum Turnaround Time : 0000
[06Eh 0110  12]        Platform ACK Register : [Generic Address Structure]
[06Eh 0110   1]                     Space ID : 00 [SystemMemory]
[06Fh 0111   1]                    Bit Width : 20
[070h 0112   1]                   Bit Offset : 00
[071h 0113   1]         Encoded Access Width : 03 [DWord Access:32]
[072h 0114   8]                      Address : 0000100001540020

[07Ah 0122   8]            ACK Preserve Mask : 0000000000000000
[082h 0130   8]               ACK Write Mask : 0000000000010001

[08Ah 0138   1]                Subtable Type : 02 [HW-Reduced Comm Subspace Type2]
[08Bh 0139   1]                       Length : 5A

Lets compare this with entries in the DSDT.

 Device (C000)
        {
            Name (_HID, "ACPI0007" /* Processor Device */)  // _HID: Hardware ID
            Name (_UID, Zero)  // _UID: Unique ID
            Method (_LPI, 0, NotSerialized)  // _LPI: Low Power Idle States
            {
                Return (PLPI) /* \_SB_.PLPI */
            }

            Name (PCPC, Package (0x17)
            {
                0x17, 
                0x03, 
                ResourceTemplate ()
                {
                    Register (PCC, 
                        0x20,               // Bit Width
                        0x00,               // Bit Offset
                        0x0000000000000000, // Address
                        0x02,               // Access Size
                        )
                }, 

What I have learned so far is that the HID (Hardware Identifier? I think) for this device is ACPI0007, which is a Processor, or Core using the earlier terminology. It has inside it a PCPC package. I have not been able to expand this acronym but I have been told that they are registers within CPPC, so possibly the letters mean the same thing but are in a different order. Maybe Processor Collaborative Performance Counter?

The two values we are interested in here are the Access Size and Address values. The Access Size is actually a zero relative index. This points to the third entry in the PCCT. This particular register has an offset of 0. The next has 0x0000000000000004, and then 0x0000000000000008, and then 0x000000000000000C, and so on.

My understanding is that in PCC there are registers and there are shared memory regions. Because a shared memory region might be off the SoC in SDRAM, it can have a lot higher latency than a register. Thus, to keep latency down, these use the on chip registers.

More on this as I learn more. And I might be WAAAAY off on this, so please do not use it to design your hardware.

when run from inside a console/ssh session will tell you the ipmi address of the machine you are on.

I’ve been working through John Madieu’s Book on Linux Device Driver Development. When typing in the Sample code for the Platform device, I got a Segmentation fault registering the device (insmod).

I started with the example code for registering the platform_driver:

static struct platform_driver mctp_pcc_driver = {
    .probe = mctp_pcc_driver_probe,
    .remove = mctp_pcc_driver_remove,
    
};
static int __init mctp_pcc_mod_init(void)
{
        int rc = 0;
        printk(KERN_INFO "initializing MCTP over PCC\n");
        rc = platform_driver_register(&mctp_pcc_driver);
        return rc;
}

The stack trace in the Kernel Oops showed that there was string compare happening in the platform_driver_register call. Looking at other Kernel modules code, I noticed that it set the .driver { .name } value. This change worked:

static struct platform_driver mctp_pcc_driver = {
    .probe = mctp_pcc_driver_probe,
    .remove = mctp_pcc_driver_remove,
    .driver = {
          .name = "mctp_pcc",
    }
};

Once I made this change, I can insmod the module, and check to see that the system sees the driver:

# find /sys/ -name mctp_pcc
/sys/kernel/btf/mctp_pcc
/sys/kernel/debug/printk/index/mctp_pcc
/sys/bus/platform/drivers/mctp_pcc
/sys/module/mctp_pcc

And rmmod-ing the module removes these /sys entries.

If you go to the Book github site and check the code, you can he he fills out the structure with the owner value as well:

static struct platform_driver mypdrv = {
    .probe      = my_pdrv_probe,
    .remove     = my_pdrv_remove,
    .driver     = {
        .name     = "platform-dummy-char",
        .owner    = THIS_MODULE,
    },
};

One other change I noticed when looking at his sample code is that he does not use an init or exit function for the module, but instead registers the driver with the single macro:

 module_platform_driver(mypdrv);

Which I confirmed also creates and removes the /sys entries upon insmod/rmmod.

When a build goes wrong, the amount of error messaging can easily scroll off the screen. Usually the error is on the first line reported. Here’s a couple ways to make it easier to read just the lines you want.

The first is just to grep for the word ‘error.’ This works for gcc:

make 2>&1 | grep error

Which produces

/root/devel/mctp_pcc/mctp_pcc.c:178:18: error: ‘name’ undeclared (first use in this function)
/root/devel/mctp_pcc/mctp_pcc.c:178:52: error: ‘fe’ undeclared (first use in this function); did you mean ‘fd’?
/root/devel/mctp_pcc/mctp_pcc.c:179:9: error: ‘ndev’ undeclared (first use in this function); did you mean ‘cdev’?
/root/devel/mctp_pcc/mctp_pcc.c:179:37: error: ‘dev’ undeclared (first use in this function); did you mean ‘cdev’?
/root/devel/mctp_pcc/mctp_pcc.c:182:17: error: ‘rc’ undeclared (first use in this function); did you mean ‘rq’?
/root/devel/mctp_pcc/mctp_pcc.c:187:20: error: ‘idx’ undeclared (first use in this function); did you mean ‘ida’?
/root/devel/mctp_pcc/mctp_pcc.c:190:24: error: ‘STATE_IDLE’ undeclared (first use in this function); did you mean ‘VTIME_IDLE’?
/root/devel/mctp_pcc/mctp_pcc.c:193:34: error: ‘mctp_serial_tx_work’ undeclared (first use in this function)
/root/devel/mctp_pcc/mctp_pcc.c:197:17: error: label ‘free_netdev’ used but not defined
/root/devel/mctp_pcc/mctp_pcc.c:183:17: error: label ‘free_ida’ used but not defined
/root/devel/mctp_pcc/mctp_pcc.c:199:1: error: no return statement in function returning non-void [-Werror=return-type]

A bit of background…all Linux processes have 3 file descriptors (FD) by default. STDIN is for reading in information from other processes. STDIN is file descriptor 0. STDOUT is for, well, STANDARD’ output, or the output that the program is supposed to produce. STDOUT is FD 1. STDERR is for output that is not standard, and is intended for error messages. STDERR is FD 2.. Make and GCC spit their output into STDERR. The trick is to tell the program to redirect standard error into STDOUT via the magic symbol 2>&1. I read this in my head as “two goes into ampersand one.” STDERR is 2, STDOUT is 1.

If you want to get just the first set of line you can use the head command like this:

make 2>&1 | head -10

Which produces

make -C /lib/modules/6.2.0+/build M=/root/devel/mctp_pcc modules
make[1]: Entering directory '/root/devel/linux'
  CC [M]  /root/devel/mctp_pcc/mctp_pcc.o
/root/devel/mctp_pcc/mctp_pcc.c: In function ‘create_mctp_pcc_nnetdev’:
/root/devel/mctp_pcc/mctp_pcc.c:178:18: error: ‘name’ undeclared (first use in this function)
  178 |         snprintf(name, sizeof(name), "mctpipcc%x", fe);
      |                  ^~~~
/root/devel/mctp_pcc/mctp_pcc.c:178:18: note: each undeclared identifier is reported only once for each function it appears in
/root/devel/mctp_pcc/mctp_pcc.c:178:52: error: ‘fe’ undeclared (first use in this function); did you mean ‘fd’?
  178 |         snprintf(name, sizeof(name), "mctpipcc%x", fe);

This gets just the first 10 lines…use a different number if you want a different amount of output

These two options are fairly easy to type and thus don’t really call for scripting. As the amount of filtering gets longer,. you make want to make scripts that build up more complex selection of the output.

If you have control of your makefile, you can use the -Wfatal-errorsflag as is written up here, but I am calling into precanned Makefiles and it drops the flag. To modify Makefiles see this discussion.

{enter} ~ .