1
0
A
0
0
0
1
1
0
1
0
D
0
0
0
0
1
1
0
1
B
0
1
0
0
0
0
1
1
E
0
0
1
0
1
1
1
0
E
0
0
1
0
1
1
1
0
F
1
1
1
1
1
1
1
1
Note that parity bits three and four are swapped. With that resolved, we can use the parity bits to decode the forward error correction, resulting in four bits being corrected, as shown in Figure 13.15. That’s LoRa!
Having reversed the protocol, it is important to look back and reflect on how and why this worked. As it turned out, being able to make assumptions and inferences about certain goings-on was crucial for bounding the problem and iteratively verifying components and solving for unknowns. Recall that by effectively canceling out interleaving and forward error correction, I was able to effectively split the problem in two. This enabled me to solve for whitening, even though “gray indexing” was unknown there were only three permutations, and with that in hand, I was able to solve for the interleaver, since FEC was understood to some extent. Just like algebra or any other scientific inquiry, it comes down to controlling your variables. By stepping through the problem methodically and making the right inferences, we were able to reduce four independent variables to one, solve for it, and then plug that back in and solve for the rest.
Figure 13.15: Forward Error Corrected bits shown in bold
Remaining Work
This paper presents a comprehensive description of the PHY, but there are a few pieces that will be filled in over time.
The LoRa PHY contains an optional header with its own checksum. I have not yet reversed the header, and the Microchip LoRa module I’ve used to generate LoRa traffic does not expose the option of disabling the header. Thus I cannot zero those bits out to calculate the whitening sequence applied to it. It should be straightforward to fill in with the correct hardware in hand.
The PHY header and service data unit/payload CRCs have not been investigated for the same reason. This should be easy to resolve through the use of a tool like CRC RevEng once the header is known.
In my experience, for demodulation purposes clock recovery has not been necessary beyond getting an accurate initial sync on the SFD. However should clock drift pose a problem, for example if transmitting longer messages or using higher spreading factors which have slower data rates/longer over-the-air transmission times, clock recovery may be desirable.
I recently published an open source GNU Radio OOT module that implements a transceiver based on this derived version of the LoRa PHY. It is presented to empower RF and security researchers to investigate this nascent protocol.29
Conclusions and Key Takeaways
Presented here is the process that resulted in a comprehensive de-construction of the LoRa PHY layer, and the details one would need to implement the protocol. Beyond that, however, is a testament to the challenges posed by red herrings (or three of them, all at once) encountered throughout the reverse engineering process. While open source intelligence and documentation can be a boon to researchers—and make no mistake, it was enormously helpful in debunking LoRa—one must remember that even the most authentic sources may sometimes lie!
Another point to take away from this is the importance of bounding problems as you solve them, including through making informed inferences in the absence of perfect information. This of course must be balanced with the first point about OSINT, is knowing when to walk away from a source. However as illustrated above, drawing appropriate conclusions proved integral to reducing and solving for each of the decoding elements within a black-box methodology.
The final thought I will leave you with is that wireless doesn’t just mean Wi-Fi anymore; it includes cellular, PANs, LPWANs, and everything in between. Monitor mode and Wireshark weren’t always a thing, so don’t take them for granted: it’s time to make the next generation of wireless networks visible to researchers, because know it or not it is already here and is here to stay.
13:8 Plumbing, not Popper; or, The Problem with STEP
by Pastor Manul Laphroaig
Gather round, neighbors. We are going to a magical place. One that we hardly ever notice in our busy lives, but which has a way of taking over your entire day when you are forced to visit it. We are going on a trip to the plumbing closet!30
Look at the miracle that is the clump of pipes, looking right back at you. Its message is clear: do not approach without skill, unless you like wet messes. This message is universal: it speaks to a politician, a professor, a columnist, an actor, and a hedge fund manager alike. It transcends languages and beliefs.
Even though these worthies and civic leaders might agree the country could use more plumbers, it has not yet occurred to them to approach the problem by putting a big P into some popular slogan like “STEP” (Science, Technology, Engineering, Plumbing), by setting up a federal Department of Plumbing, or by lionizing a professional TV personality who goes by “A Plumbing Guy,” despite never having fixed a pipe in his life.
They somehow know that these things will do diddly squat to address the shortage of plumbers. They know deep down that to learn plumbing—and even to not sound ridiculous about it—one needs to study with a plumber, attach oneself to a plumber, and do what a plumber does for a while. This, neighbors, is how deep the plumbing magic goes.
Science, alas, has not been so lucky.
It is fashionable to talk about how we need more scientists, and how we can direct and improve science, quoting grand theories that explain science, while similarly educated people nod approvingly. After all, they all know what science is, as befits all forward-thinking people these days. No one feels awkward; everyone feels good.
Perhaps this happens because our social betters all experienced helplessness at the sight of broken plumbing, but would not recognize broken science, much less a hopelessly broken science textbook. You see, science lab equipment is OK with a patronizing, self-satisfied gaze, whereas plumbing has a way of glaring back contemptuously, daring you to use your general theoretical understanding.
With plumbing, it’s either practical skill or a huge mess in your basement. Messing with how plumbers learn and teach this skill guarantees messes in thousands of basements. If you value your plumbing, it’s wise to leave plumbers alone even if you believe every word of every newspaper column you’ve ever read on plumbing economy.
It may be a surprise to the readers of Karl Popper and Imre Lakatos that actual scientists are helped by philosophy of science in exactly the same way as plumbers are helped by the Zen of Plumbing.31 Although these very same people are likely to believe they understand plumbing too, they usually have the sense to leave the plumbing profession well alone, and not apply their philosophical understandings to it—being empirically familiar with the fact that when you need plumbing done, philosophy is useless; only skill stands between the water in your pipes and your expensive library.
By far the worst hit to a profession is delivered when a part of the professionals actually welcomes philosophers lauding it, politicians bearing gifts and grants, and governments setting up departments to promote it. Forms to fill, ever-growing grant application paperwork, pervasive “performance metrics,” and having to exp
lain basic fallacies to the well-meaning but fundamentally ignorant and hugely powerful committees come later—and accumulate. In the context of metrics, charlatans always win, because they don’t get distracted by trying for actual results.
Not to mention that the money that goes to charlatans is not net-neutral for actual plumbing (or science); it is net-negative, because charlatans have a way of making the lives of professionals hard where it hurts the most. When Tim “the Tool Man” Taylor waves power tools around with a swagger, the results are immediate and obvious. When learned committees do the professional equivalent thereof to math textbooks and call it nice names like “Discovery Math,” “Common Core,” or “Critical Thinking” it takes a generation to notice, and then we wonder—how on earth did school math become unteachable and unlearnable?32
Plumbers have wisely avoided it, perhaps due to some secret wisdom passed from master to apprentice through the ages. Scientists, I am sorry to say, walked right into it around the middle of the twentieth century.
Sure enough, national agencies got us to the moon—but it seems that all the good science schoolbooks have been put on the rockets going there, never to return. Have you met many scientists who are happy with what schools do to their sciences after half a century of being improved by various government offices?
Funny how it worked out for scientists. Now hear them complain about “publish or perish,” the rapidly rising age at which one finally succeeds in getting one’s first grant, and the relentless race to rebrand and follow the current big-ticket grant programs.33
But don’t blame them, neighbors; it was their advisors or their advisors’ advisors who fell for it. Better to buy them a drink, and remember their lesson.
Better yet, find some plumbers, and buy them drinks. Perhaps they’ll share with you some of their secrets of how to keep the philosophers and their educated and benevolent readers interested in the result, but at a safe distance from the actual plumbing.
13:9 Where is ShimDBC.exe?
by Geoff Chappell
Microsoft’s Shim Database Compiler might be a legend ... except that nobody seems ever to have made any story of it. It might be mythical ... except that it actually does exist. Indeed, it has been around for fifteen years in more or less plain sight. Yet if you ask Google to search the Internet for occurrences of shimdbc, and especially for “shimdbc.exe” in quotes, you get either remarkably little or a tantalising hint, depending on your perspective.
Mostly, you get those scam sites that have prepared a page for seemingly every executable that has ever existed and can fix it for you if only you will please download their repair tool. But amongst this dross is a page from Microsoft’s TechNet site. Google excerpts that “QFixApp uses the support utility ShimDBC.exe to test the group of selected fixes.” Follow the link and you get to one of those relatively extensive pages that Microsoft sometimes writes to sketch a new feature for system administrators and advanced users, if not also to pat themselves on the back for the great new work. This page from 2001 is titled Windows XP Application Compatibility Technologies.34
Application Compatibility?
There can’t be anything more boring in the whole of Windows, you may think. I certainly used to, and might still for applications if I cared enough, but Windows 8 brought Application Compatibility to kernel mode in a whole new way, and this I do care about.
The integrity of any kernel-mode driver that you or I write nowadays depends on what anyone else, well-meaning or not, can get into the DRVMAIN.SDB file in the AppPatch subdirectory of the Windows installation. This particular Shim Database file exists in earlier Windows versions too, but only to list drivers that the kernel is not to load. If you’re the writer of a driver, there’s nothing you can do at run-time about your driver being blocked from loading, and in some sense you’re not even affected: you’re not loaded and that’s that. Starting with Windows 8, however, the DRVMAIN.SDB file defines the installed shim providers and either the registry or the file can associate your driver with one or more of these defined shim providers. When your driver gets loaded, the applicable shim providers get loaded too, if they are not already, and before long your driver’s image in memory has been patched, both for how it calls out through its Import Address Table and how it gets called, e.g., to handle I/O requests.
In this brave new world, is your driver really your driver? You might hope that Microsoft would at least give you the tools to find out, if only so that you can establish that a reported problem with your driver really is with your driver. After all, for the analogous shimming, patching, and whatever of applications, Microsoft has long provided an Application Compatibility Toolkit (ACT), recently re-branded as the Windows Assessment and Deployment Kit (ADK). The plausible thoroughness of this kit’s Compatibility Administrator in presenting a tree view of the details is much of the reason that I, for one, regarded the topic as offering, at best, slim pickings for research. For the driver database, however, this kit does nothing—well, except to leave me thinking that the SDB file format and the API support through which SDB files get interpreted, created, and might be edited, are now questions I should want to answer for myself rather than imagine they’ve already been answered well by whoever managed somehow to care about Application Compatibility all along.
The SDB File Format
Relax! I’m not taking you to the depths of Application Compatibility, not even just for what’s specific to driver shims. Our topic here is reverse engineering. Now that you know what these SDB files are and why we might care to know what’s in them, I expect that if you have no interest at all in Application Compatibility, you can treat this part of this article as using SDB files just as an example for some general concerns about how we present reverse-engineered file formats. (And please don’t skip ahead, but I promise that the final part is pretty much nothing but ugly hackery.)
Let’s work even more specifically with just one example of an SDB file, shown in Figure 13.16. It’s a little long, despite being nearly minimal. It defines one driver shim but no drivers to which this shim is to be applied.
Although Microsoft has not documented the SDB file format, Microsoft has documented a selection of API functions that work with SDB files, which is in some ways preferable. Perhaps by looking at these functions researchers and reverse engineers have come to know at least something of the file format, as evidenced by various tools they have published which interpret SDB files one way or another, typically as XML.
As a rough summary, an SDB file has a 3-dword header, for a major version, minor version, and signature, and the rest of the file is a list of variable-size tags which each have three parts:
a 16-bit TAG, whose numerical value tells of the tag’s type and purpose;
a size in bytes, which can be given explicitly as a dword or may be implied by the high four bits of the TAG;
Figure 13.16: ShimDB File
and then that many bytes of data, whose interpretation depends on the TAG.
Importantly for the power of the file format, the data for some tags (the ones whose high four bits are 7) is itself a list of tags. From this summary and a few details about the recognised TAG values, the implied sizes and the general interpretation of the data, e.g., as word, dword, binary, or Unicode string—all of which can be gleaned from Microsoft’s admittedly terse documentation of those API functions—you might think to reorganise the raw dump so that it retains every byte but more conveniently shows the hierarchy of tags, each with their TAG and size if explicit or data if present. A decoding of Figure 13.16 is shown in Figure 13.17.
To manually verify that everything in the file is exactly as it should be, there is perhaps no better representation to work from than one that retains every byte. In practice, though, you’ll want some interpretation. Indeed, the dump above does this already for the tags whose high four bits are 6. The data for any such tag is a string reference, specifically the offset of a 0x8801 tag within the 0x7801 tag (at offset 0x0142 in this example), a
nd an automated dump can save you a little trouble by showing the offset’s conversion to the string. Since those numbers for tags soon become tedious, you may prefer to name them. The names that Microsoft uses in its programming are documented for the hundred or so tags that were defined ten years ago for Windows Vista. All tags, documented or not (and now running to 260), have friendly names that can be obtained from the API function SdbTagToString. If you haven’t suspected all along that Microsoft prepares SDB files from XML input, then you’ll likely take “tag” as a hint to represent an SDB file’s tags as XML tags. And this, give or take, is where some of the dumping tools you can find on the Internet leave things, such as in Figure 13.18.
Figure 13.17: ShimDB File (Decoded from Figure 13.16)
Notice already that choices are made about what to show and how. If you don’t show the offset in bytes that each XML tag has as an SDB tag in the original SDB file, then you risk complicating your presentation of data, as with the string references, whose interpretation depends on those file offsets. But show the offsets and your XML quickly looks messy. Once your editorial choices go so far that you don’t reproduce every byte but instead build more and more interpretation into the XML, why show every tag? Notably, the string table that’s the data for tag 0x7801 (TAG_STRINGTABLE) and the indexes that are the data for tag 0x7802 (TAG_INDEXES) must be generated automatically from the data for tag 0x7001 (TAG_DATABASE) such that the last may be all you want to bother with. Observe that for any tag that has children, the subtags that don’t have children come first, and perhaps you’ll plumb for a different style of XML in which each tag that has no child tags is represented as an attribute and value, e.g.,
PoC or GTFO, Volume 2 Page 40