Don’t Support the Coreboot Project
Table of Contents
UPDATE: We set out with the goal of offering open-source firmware on our laptops, potentially becoming one of the few companies worldwide to do so.
Despite our willingness to pay fair market rates, we encountered significant resistance from two coreboot consultants and one coreboot leader.
The current coreboot documentation is inadequate, forcing many companies to rely on consultants. Our experience demonstrates the problems with this approach.
Had we succeeded, we planned to improve coreboot’s documentation to make it easier for other companies to port coreboot to their boards.
The actions of these three coreboot developers not only prevented us from offering open-source firmware but also hindered potential improvements to the coreboot ecosystem.
The coreboot project has lost both a potential hardware provider and valuable documentation contributions.
We share this update to provide context for our article and to highlight the broader implications of our experience for the open-source community.
__________
UPDATE 2: System76’s Principal Engineer decided to chime in and make a fool of himself, so we banned the entire state of Colorado for life.
__________
UPDATE 3: We haven’t read coreboot’s response and never will. That project is trash and doesn’t deserve anyone’s attention.
If they claimed our account of what happened is in any way false, they are lying. God can strike me dead if anything we said isn’t true or if we left out a single relevant fact or detail.
What these coreboot consultants and leaders did is simply indefensible. Anyone who sides with them should hang their head in shame.
The coreboot project is a cancer and a debt to the entire enterprise of open source and should be abandoned.
__________
Have you ever wondered why so few companies offer laptops with coreboot? We found ourselves asking the same question, which led us on a 15-month journey to port this open-source firmware to our laptops. The experience, however, was nothing like we anticipated.
Instead of support and collaboration, we faced inflated prices, poor communication, and what we perceived as incompetence from coreboot leaders and consultants. Our efforts ultimately fell apart, not due to technical challenges, but because of what we see as unethical practices within the coreboot community.
This article details our interactions with three different coreboot contributors: two consultants and one leader. We’re sharing our experience to expose the issues we faced and to question why it’s so difficult for companies to adopt open-source firmware.
9Elements (https://www.9elements.com)
We originally reached out to 9Elements last year along with several other coreboot consultants, but all of their prices were so outrageous ($50k-$100K per board) that we decided to try porting coreboot to our laptops ourselves. After hitting a sticking point, we reluctantly contacted them again for help debugging our code.
From the start, our interactions with Christian Walter were awful. We repeatedly stressed how important and time-sensitive this project was, but he seemed completely indifferent. In fact, he made a snide remark about us coming back after trying to do it ourselves.
If we didn’t feel like we needed a coreboot specialist to complete the porting in a timely manner, we would have never hired 9Elements based on the price and Christian’s insufferable attitude and poor communication skills.
A lot of Christian’s emails looked like they were written by a 12-year-old on their phone. He didn’t capitalize the first letter of a lot of sentences, had obvious typos, and improper punctuation. He also had the habit of talking down to us, repeatedly trying to “put us in our place” and “set us straight” when we were just trying to get help debugging our code. I guess this is the level of service you get for $300 an hour in Germany.
When quoting the project, Christian tried to undermine our existing work, saying, “The coreboot code is very incomplete: There is basically nothing expect the device tree and the GPIO config.” In reality, we actually had everything needed to build the coreboot ROM.
He quoted 20 days (160 hours total), which assumed we had no code at all. Here is a direct quote from Christian, “And yes – the 20 days would be only coreboot + EDKII – that would be only an assumption. I jsut want to make you aware of that it will be much more work then “just debugging”.”
After we mentioned that we wanted to do as much as we could ourselves to keep costs down, Christian even tried to make a deal where 9Elements would receive a commission from every laptop we sold. He said, “I think I suggested an alternative before: Trying to form an agreement that we get a share of every sales, and we will support the device at a low cost price, I think you did not respond by the time being on my mail there.”
Yes, giving 9Elements a commission on every laptop we sold certainly sounds like a great way to have kept the costs down. This is really how shameless they are in trying to take you for every penny you’re worth. You’ll see what I mean when you find out the actual scope of the work required for this job below.
After reviewing our schematics and motherboard, Christian said, “There are not debugging capabilities on the board.” We had to actually inform him that there is, in fact, a UART port on the motherboard for debugging purposes.
This is just one example of how Christian didn’t seem to really know what he was doing, or how we had to point out something that you wouldn’t assume a professional firmware developer would need help with.
Another aspect we were dissatisfied with was the developer we were assigned. Considering how much we were paying, we assumed we would get one of their top coreboot contributors. Instead, we got an inexperienced developer named Max, who, according to his LinkedIn, was only two years out of college. Christian even admitted that Max had only worked on a couple of other coreboot-related jobs. So much for paying a premium to work with a coreboot specialist.
Weeks elapsed with little to no activity, because they were super busy pretending to be doing something else out in the abyss of phantom world. Eventually, Christian claimed that Max was working on our project full-time. However, upon chatting with Max on Slack, he admitted that he was only in the office for a few hours each day.
Several more days passed without any updates. When we finally asked Max for one, he said he was having issues reading and writing to the chip. At this point, we were extremely frustrated. It had been two months since we first contacted them. Christian claimed Max was working full-time, but that clearly wasn’t the case. Max was relatively inexperienced and didn’t seem to know what he was doing. Given all this, we asked Christian if we could get a second developer to work with Max until they resolved the issue.
Well, much like everything else, Christian took offense to us asking for a second developer, which turned out to be the beginning of the end.
This led to a private chat with Christian, where we questioned him about Max stating he was only in the office a few hours a day. We said we thought full-time meant 8 hours a day, 5 days a week, and Christian confirmed that was what he meant by full-time.
As we were having this chat, Max made a breakthrough, thanks to the advice we gave him about what to check to resolve the reading and writing issue. He finally built the coreboot ROM with our code, flashed it, and tried to boot the laptop, which displayed an FSP message. Max said he was surprised it made it that far. Why? We told them our code just needed debugging, but they didn’t want to believe it.
Despite the good news, Christian continued talking to us like dirt in the private chat, so we asked him if he was the owner of 9Elements, because an employee would never talk to a customer like this, and he confirmed he was. Then we said, “We can tell because owners usually aren’t the best at customer service.” To which he replied, “I understand.”
And then, within just a couple of minutes and without any warning, after having invested two months into trying to work with them and just 15 minutes after Max figured out how to read and write to the chip, Christian removed us from the Slack channel we were invited to and essentially told us to get lost. We even had to remind him that he still had our laptop, and we wanted it back.
Due to this experience with 9Elements, we banned the entire country of Germany for life.
3mdeb (https://www.3mdeb.com)
3mdeb is the next coreboot consultant we tried working with. Throughout our interactions with them, we encountered delays, inflexibility, and a stark contrast between their open-source rhetoric and their actual business practices.
3mdeb tries to market itself as a champion of free and open-source software by highlighting its values on their site, but what open source really seems to be about for them is extracting all of the value from other people’s code and then repackaging it under their own brand and charging a small fortune for it.
Despite our willingness to work collaboratively and our existing progress on the project, 3mdeb insisted on their full, expensive porting process, seemingly prioritizing their brand and profit margins over actually advancing open-source firmware adoption.
They wanted $3,300 for a Dasharo-branded coreboot + EDK II port evaluation and $6,600 for an unbranded coreboot + EDK II port evaluation, which would accomplish little more than giving us a quote for the actual porting itself.
They issued this quote when they thought we wanted coreboot ported to two laptops. After we told them it was only one laptop, they resent the quote and the pricing was exactly the same for one laptop as for two laptops, which kind of leads you to believe there is no real evaluation, otherwise, two laptops would cost twice as much as one laptop.
Here is what they said, “Our Dasharo services pricing is based on the complexity and scope of the project, not the number of laptops.”
We never received a quote for the actual porting, but they said the evaluation cost is typically 10% of the total cost, which would mean the porting would have cost around $33,000 for Dasharo-branded coreboot, and $66,000 for unbranded coreboot. Even their highly discounted Dasharo-branded porting comes out to around $250 to $330 an hour, and that’s if they started from scratch. We had 80%+ of the job already complete. We just needed to debug our code.
The company’s decision to charge significantly more for unbranded coreboot firmware, despite it being identical in code to the less expensive Dasharo-branded version, demonstrates a priority to promote their Dasharo brand over supporting the broader adoption and development of open-source firmware.
This pricing strategy raises several ethical concerns:
- Artificial barriers: By making unbranded coreboot significantly more expensive, 3mdeb is creating artificial barriers to its adoption, potentially limiting innovation and collaboration within the wider coreboot community.
- Misalignment with open-source principles: This approach seems to contradict the core values of open-source software, which emphasize accessibility, collaboration, and community-driven development.
- Profit over progress: It suggests that 3mdeb is more interested in monetizing and controlling the use of open-source software than in fostering its growth and widespread adoption.
- Lack of transparency: Customers may not be fully aware that they’re paying a premium solely for the absence of Dasharo branding, despite receiving functionally identical software.
- Potential conflict of interest: As contributors to the coreboot project, 3mdeb’s practices could be seen as exploiting their position within the open-source community for financial gain.
- Discouraging customization: This pricing model may discourage companies from developing their own branded firmware solutions, potentially stifling innovation in the field.
So, after some back and forth on the pricing, we made this offer, “We will pay $150 an hour for this service until the project is complete. I would have direct access to the developer via Slack or a similar messaging platform, so I can assist them and do daily reviews of their work. This means no evaluation stage; we just start working, and if you determine you cannot complete the project at any point then you still get paid for whatever work you performed, and we will prepay in $1500 increments, so it’s like buying one of these 10-hour blocks of the service you provide at the link above.”
3mdeb responded by saying, “The support packages offered in our online shop are for Zarhus (system) and Dasharo (firmware) ecosystem support. We provide custom support based on custom mutual agreements with our clients, not on an hourly basis for complex firmware development projects. Custom firmware development and body leasing mean higher rates, as outlined in our offer.”
Remember, their Dasharo firmware is just a rebranding of coreboot + EDK II. So it’s total nonsense for them to offer the pricing for their Dasharo-branded version of coreboot but not for unbranded coreboot, as the work required for our project is identical regardless of how the firmware is branded.
And just to put this all into perspective, the average hourly wage of a firmware developer in the US is $75, and in Poland, where 3mdeb is located, it’s $25. When adjusted for cost of living, $150 an hour in Poland is equivalent to making $300-$375 an hour in the US.
But, apparently, this wasn’t enough for them to debug open-source code for one of the main projects they are contributors to. Does that seem real to you? Because I’m still wondering if I dreamed all of this.
Due to this experience with 3mdeb, we banned the entire country of Poland for life.
Complaint
At this point, we decided to file a complaint with the coreboot leadership team to see if they could do anything to help us resolve the issues we were having with their consultants.
David Henricks, one of the coreboot leaders, replied and said, “We cannot dictate terms to consultants, other than that they must actually be able to support coreboot and use the code and trademark appropriately.”
Great to know, Dave. I assumed coreboot could stop endorsing a company or recommending them on their site if they treat companies looking to port coreboot to their hardware like trash or engage in unethical business practices, but apparently not!
Then another coreboot leader, Matt DeVillier, replied, and we actually came to an agreement for him to help us complete the porting. Needless to say, we were very excited about the fact that Matt agreed to work with us.
Since he was on the leadership team, we knew he wouldn’t treat us like the coreboot consultants did, and being an individual and not a business meant there would be less bureaucracy as we could communicate directly with him, which would allow us to work more efficiently.
And since we already had the pricing agreed upon, we felt like we were finally going to be able to complete the porting of coreboot to our laptop.
MrChromeBox (https://www.mrchromebox.tech)
Matt DeVillier is the type of person who will suck your will to live out of you and leave you for dead. Nice guy. He embodies every trait you would never want in someone you hire for a job. He’s unprofessional, unreliable, untrustworthy, irresponsible, and incompetent.
And, of course, he’s one of these types who will bludgeon you with all of these virtuous qualities, and if at any point you dare say a word in protest, he’ll try to turn things around and act like you did something wrong by calling you toxic. No, we’re not toxic, Matt. You just really honestly truly suck.
If everyone in the world was like Matt DeVillier, I’m not even sure how it would continue to function. Who in their right mind would ever employ such a person? Well, apparently, AMD, and that’s why we banned all of their products for life, and we had plans to offer an AMD-based version of our next-generation of Aon laptops.
The day he received the laptop we sent him, he said, “yes, arrived late this afternoon. hope to have some time to check it out tonight.” One day later, up until which point he maintained, “Right now I’d only be able to put in 10hrs/week on average,” he said, “So unfortunately, I just got out of a meeting where I’ve been assigned to a very high priority project for the next ~2 months, and I’m going to have zero time to work on any side projects like this until mid/late August at the earliest.”
Is Matt a firmware developer or an ER surgeon? Because I had no idea the life of a firmware developer was so exciting and unpredictable that you don’t know what your schedule will be literally from one day to the next.
Is this standard operating procedure at AMD? They have emergency firmware projects where their employees have to work overtime for a couple of months? Because you would assume this high priority project must mean he’s working extra hours; otherwise, why would it prohibit him from working on our project when he already said he could dedicate an average of 10 hours a week to it?
Regardless of AMD’s project management style, it’s clear that Matt’s urgent responsibilities didn’t exactly curtail his other side projects. Strangely enough, his GitHub activity spiked in June and July—the very months he was meant to begin our project—more so than at any point in the previous six months.
So, while we were waiting for him to start, it came to light that he didn’t even have the tools needed for the job, even though we discussed this in detail before he received the laptop, and he clearly said he had everything we told him he needed.
He sourced one of the tools directly from China (because, apparently, Matt can’t do anything right), and it got stuck in customs for a month. At which point, he decided to source another one from the US, because, as it turns out, we have basic firmware tools available here.
After he started to work on the laptop, he said a cable ripped while removing the motherboard. Did he apologize? No. Did he offer to pay to fix it? No. Like a true leader, he immediately tried to place blame on someone else by saying whoever had the laptop before him manhandled it.
Here is a direct quote from Matt, “Though someone that had this laptop before me made a fucking mess of the UART connector, I’m not sure I can clean it up and make it usable.”
The funny thing is that the only other party besides him who had touched what was otherwise a brand-new laptop was one of the coreboot consultants mentioned above, 9Elements. So, if anyone manhandled it, it was them.
As the process dragged on, with Matt giving us sporadic updates, he said he was having trouble reading and writing to the chip, and it was because he didn’t have a sufficient power supply, which we also discussed before the laptop arrived and he said wouldn’t be a problem. Do you notice a pattern?
So, after almost 10 weeks of accomplishing literally nothing other than damaging our laptop, we asked for an ETA on when he might officially begin and finish the project, since he had only been tinkering up until that point.
He asked us for our code, which we had already sent to him the day after we first contacted him, in response to him saying he wanted to evaluate our code before committing to the project, so we resent it and followed up again on the status, but still didn’t get a reply.
After waiting several more days, we wrote this one-line email, “What’s going on over there? You’ve left us high and dry, and we have no idea what to even tell our customers at this point.” Matt replied, “You’d be better off finding someone else at this point.” Essentially, he was telling us he wanted to quit the project.
After some back and forth, he maintained that he hadn’t done anything wrong by abandoning the project after nearly 10 weeks, so we offered to arbitrate the matter and let a neutral third party decide, but Matt refused to engage in this process, saying, “I’m not interested in litigating this with you via email or otherwise,” and why would he when all he had done was damage our laptop and waste 2.5 months of our time.
And just to emphasize how little character Matt has, at the end after it was determined he wouldn’t be helping us port coreboot to our laptop, he actually said he never even agreed to help us. Here are his exact words, “I never committed to either starting or completing this project contractually, orally nor in writing.”
But yet when we first contacted him about this project, he asked us many detailed questions about the scope of the work and then in response to our answers said, “Right now I’d only be able to put in 10hrs/week on average.” And then, “I’m pretty sure I can commit to at least getting coreboot working, but let me eval your current codebase and boot log before committing.”
And then we sent him our code, and he agreed to let us send him a laptop to work on. Based on this, under what circumstances was he not going to help us port coreboot to our laptop?
If he would have completed the job, is his position that we wouldn’t have owed him anything even though during the initial email exchange we said we would pay him $150 an hour for the work? Of course not.
After he quit, we asked Matt to pay for the cable he destroyed and he refused, saying he didn’t rip it in half. His exact words were, “Any hardware damage was present on the board when I received it (which I communicated to MALIBAL at the time). To assert that any damage was caused by me when MALIBAL has not even yet received back the laptop demonstrates this communication, the veracity of my statement, and the ridiculousness of your assertion.”
In reality, this is what he actually communicated to us when he informed us about the ripped cable, “Also, one of the ribbon cables on the mainboard ripped when removing the mainboard to solder the wires to the debug port. I suspect given the condition of the device it was previously manhandled and already on the cusp of tearing.”
This is the type of gaslighting you would only expect from a sociopath. Contrary to him saying it was ridiculous to claim he caused any damage to our laptop; you can see he clearly admitted to damaging a cable when removing our laptop’s motherboard.
And when you really think about it, could a literal monkey have damaged the cable any worse when removing the motherboard than completely ripping it in half? This is what you might call being maximally bad. That’s where you do such a horrible job that it’s impossible to have done any worse. Not impossible for a person to have done worse; impossible for anyone or anything to have done worse: a person, a monkey, a rock, etc.
The cherry on top of all of this is that, unlike with the coreboot consultants, we told Matt that this was our last attempt to port coreboot to our laptop, and that if he didn’t help us we would never offer coreboot on our laptops, and he decided to quit anyway, knowing it would also mean we lost all of the work we put in over the span of 15 months.
Due to this experience with Matt DeVillier, we banned the entire state of Maryland for life.
Commentary
If we had successfully ported coreboot to our laptops, we had planned to share our experience by informing other companies about the actual time it takes and how to do it themselves, so they wouldn’t have to endure the same incompetence and cartel-like dealings we experienced with these coreboot leaders and consultants.
Based on everything we learned about coreboot, we think you can port coreboot to a laptop board in 40 hours or less. There are tools that you can use to dump most of the code you need. For instance, the GPIO can be done in 30 minutes, as opposed to the 30-50 hours that was quoted by several developers.
The data.vbt file can be dumped in 30 seconds. Jay Talbott from SysPro Consulting, another coreboot consultant who we contacted, said he didn’t know you could do this. He said, “We’ve always started from the example VBT from Intel for their reference board, and modified it with BCT or DisCon as needed to align with the target board design.” No telling how many hours that takes. We sent him the data.vbt file that we dumped, and he said it looked like an actual VBT binary and would probably work.
There is another company that seemingly supports ~10 new laptop boards every year, and they are surely not spending 1000 to 1500 hours (100-150 hours per laptop) a year doing it, which is what convinces me that it can be done in much less time. And if so, that shows you just how insanely egregious the price gouging really is. That would make the hourly rate come out to between $825 and $2500 based on the various fixed price quotes we received from the coreboot consultants.
Closing
In the 6+ months that we tried working with these three coreboot developers, they didn’t write or debug a single line of code. Is that believable? No, it’s not.
And for the record, in all of our correspondence with the coreboot team, we never called anyone a monkey nor implied they were one, nor did we make any other overtly condescending remarks or insinuations. We used that sort of tone in this article to simply illustrate a point.
If we would have said something to the effect of, “We felt the experience was unsatisfactory and thought they could have done a better job…” it would not have in any way conveyed the magnitude of the actual emotional distress that we were subjected to, which can be proportionally measured in our response to ban their states and countries.
In all of our correspondence with these coreboot developers, we went out of our way to get along. But despite our best efforts, and putting up with all of their nonsense, we were ultimately thrown to the curb like trash.
Which is why anyone who supports the coreboot project after the post date on this article, including contributing code, being a member of their boards, or donating will be banned for life – without exception.
If you want to keep up to date on our future plans for developing open-source firmware for our laptops, please subscribe to our RSS feed: https://www.malibal.com/feed/