Meltdown and Spectre – the hidden problem


For those of you that follow me on social media, you will know that the last week I’ve been publically busy dealing with the fallout from the discovery by Google of the flaws in the Intel CPU design that have led to many of us being extremely busy.

The press coverage in the main is sensationalist but in a way useful. It tells a story of a major company (Intel) who didn’t ever think about the fact separation in the kernel was a good thing.

More it speaks volumes where many believe a major company, Google could have been viewed potentially to have played commercial advantage to seek an embargo that ran from discovery in mid-2017 to the 9th of January. I’m not sure I agree entirely with that given the investment Google has in kit powered by Intel across it’s Cloud and Waymo estate, plus the Chromebook market, but I understand why that would appear to some to be an advantage.

The issue is the delay in the lead time that did not include timely bringing in the key players who would be able to generate mainstream fixes for business as usual compute needs. Now we can only speculate why that was that the likes of Red Hat and Canonical (who are still at the RC kernel phase for some platforms) were not brought in until November but it’s not a good look for the management of either Google or Intel.

In fact, it stinks. When you have a bug this massive that potentially affects the mainstream OS’s you talk to the vendors as soon as you have a clear idea. Not three months later. I’m going to tell you why and you won’t see it coming.

Just so that this is very clear, I am writing this as me, a technology journalist who writes for other outlets, a security guy who has invented and released stuff that protects big swathes of the worlds educational and commercial computing fabric. But I’m also writing this as a member of the Open Source community for two-plus decades and a former Red Hat security staffer in my own words without influence.

So the security industry I have chosen to work in since we wrote SmoothWall in August 2000 is made up of key players in software, hardware and services. The software side is the small part of the industry, the majority of key revenues are tin, companies buying major chunks of hardware in the firewall, DNS, load balancing, gateways, VPN hardware, specialist hardware for authentication and key encryption (more on that later) and lots more besides. Trade shows such as Infosec and RSA are kept afloat by the vendor stands packed full of racks of equipment.

All of this kit from hundreds of vendors, has a lifecycle. Now here is a statement of fact.

If you look at the demographic of many of these security vendors they survive by evolution of kit and tech refresh demands of customers. If you delve deeper you see that the majority of their effort goes into sales and marketing of this kit, closely followed by customer services and maybe return to base support in that order. The development environments are tiny. The number of developers on staff at most security hardware vendors is small by comparison to the number of folk involved in the commercial side of the business. It’s not unusual to see a security vendor where more people work in HR there than work in the brains of the software business.

The software that runs on these devices is predominantly Linux, sometimes BSD, but normally Linux and normally its a version of CentOS (although there are some Ubuntu rack based devices out there). CentOS is a derivative of Red Hat that I am closely associated with and that follows the Red Hat Enterprise Linux (RHEL) pathway. E.g they have a live 6.x and 7.x and EPEL release cycle. CentOS is not RHEL. CentOS and RHEL do not share a kernel. CentOS is used in hundreds of millions of deployments daily globally.

If CentOS did not exist the likes of Facebook and eBay, CERN and GoDaddy would have a problem, you don’t see those organisations ponying up to Red Hat to part with cash, like many others they shun RHEL to use CentOS which they see as “like enough” to stand up mission critical platforms. More importantly, they support themselves with intelligent capable engineering staff capable of standing up repositories and dealing with day to day proactive support.

For everyone else, there is Red Hat RHEL supported world-class Linux backed up by QE, backed up by amazing support staff and with a legacy history of being best in class. CentOS maintainers as a rule since a few years back work at Red Hat and we all respect each other hugely and count each other as friends. But let me repeat, CentOS is not RHEL, even if they do release the patches and RPMs that Red Hat release once Red Hat has put the QE and massive security patching that my former team get out the door.

Now we’ve got that straight lets work out why this is a bad thing for the device market, and potentially for the entire security market as we understand the longterm issues surrounding Meltdown.

We made the point already that many of the security vendors have small dev teams. Many security vendors making tin go bust, many get swallowed up by other vendors. The one common thing is that security kit in the field whether the vendor has gone bust or been acquired or is still trading is running a Linux derivative on an Intel chassis, some Xeon or Haswell chipset or any of the thirty-plus derivatives of each going back seven plus years.

Many (a lot) of these devices are still running platforms that started out in the development lab at the vendor as CentOS 4/5/6/7 development trees. For the later versions thats fine and dandy, kernel and microcode patches are available due to CentOS benefitting from the hard work Red Hat did to get the patches out for a multitude of architectures. Hat tip to my amazing friend and brother-in-arms Cliff Perry for having lost a lot of Christmas dealing with this so capably, and assisted by his team in Brno and the engineering team in Westford.

However, a lot of the devices are running versions 4 and 5 and have long since departed from being “standard builds”. And theres a reason for this and it’s not one you’ll notice straight away because it’s utterly non obvious.

Many of the Linux based tools out there that run older versions of CentOS 4 and 5 from big name vendors run older versions of Samba the CIFS tool we developed to allow Linux to sit in heterogenous Windows environments. The older 2.x version of Samba being licenced under GPL version 2. In Samba versions 3 onwards changes in the GPL licencing meant that the licencing and patents issue reared it’s head. I’m not going to go into details as most people reading this immediately get it and understand why it meant for major vendors building tin relying on Samba / Winbind / basic Active Directory authentication it meant potential loss of IP. You can read more on the differences between GPL v2 and v3 here so that I don’t need to go into detail.

If you are a vendor deploying a Linux based device using GPL code you are supposed to have on your website somewhere, or even ship with your device as many vendors do, a copy of the GPL and make applicable modified sources available. TP-Link, D-Link, Netgear and other vendors understand that their reliance entirely on the work of the Open Source community and Linux as a whole makes them understand their lineage and do just that.

There are many vendors in the security space who harness large amounts of Linux as the base OS and base development environment who do not. That gripe, that will have to wait for another day to moan about as it’s only a side effect it’s not the gift that keeps on giving that will keep security folk on their toes for the next 3-5 years.

No the bigger issue is that there are major vendors out there with devices empowering large chunks of the internet estate and cloud estate we rely on with deployed racked kit running CentOS versions 4 and 5. Either because they have dependencies on libraries and tools they’ve developed or their IP which is compiled against those kernels, or the need to run pre GPL ver 2 Samba variants and supported dependencies.  The list of those vendors includes many of the household names in hardware that you see at many security trade shows.

Traditionally those vendors would prefer that customers did not treat their devices with the same duty of care as they would, for instance, a RHEL/CentOS server in production running NoSQL or as a web server. No they’d prefer you treated it as an appliance. The fact that both that production server and the appliance are racked 1U apart and connected to the same switch is not important to the vendor, they just want you to remember – they ship an appliance, not a server.

If you were to draw a chart and look at the updates applied to the server and to “the appliance” that affected actual computing security needs (we are not talking spurious non-mission critical updates with no dependency) to both boxes over time it would be illuminating. Illuminating because you’d see the server would receive regular kernel updates, regular updates to OpenSSH, OpenSSL etc. If you looked at the appliance over the same period in time the number of updates that were applied would be a lot smaller, appliance vendors traditionally being a lot weaker. Also, they have staff who although supportive of kit still under lifecycle support are tasked with writing next generation lifeblood for new kit under development. 

So we have appliances and we have servers. And for some reason, we are supposed to treat them differently. Both have privileged users, both are based on Linux, but we’re supposed to treat one with care as a server as it’s performing a task that is mission critical and we’re supposed to treat the other as a bit of tin (that’s also performing a task that is mission critical). 

Here is the problem. A huge chunk of our security estate is built out on non supported non patchable variants of CentOS and other Linux variants. Those devices are authenticated on our networks, many have small to medium amounts of storage on board, many of them you can get a shell on. Many of these devices are end of life and still in use in many organisations who haven’t removed them at tech refresh because they still work and are the glue they require, and if it ain’t broke why fix it.

All of them run Intel hardware. 

Eighteen months ago when I was still at Red Hat running security strategy I built a plan to go and see the vendors to get them to stop using non supported CentOS and to use Red Hat Enterprise Linux as their base because it gave them seven to ten years support for shipping binaries so situations like this couldn’t happen. I left before I got a chance to do it. We had identified the gap and the scale of the issue and it was enormous. I’m sorry I never got the chance to but other opportunities came up and after seven years at the helm I had a chance to exit and took it. 

Meltdown and Spectre. I’m not overly interested in the patching of workstations and servers, for me as a security guy I’m interested in the glue that holds this all together. The fact is that major chunks of the internet estate are now glued together using non patchable kit. 

For those of us in security monitoring it’s manna from heaven, corporates of all sizes are affected massively and now have to deal with it as their vendors will not be able to release timely patches to secure architecture. There is literally no alternative if you’re made aware that your estate is at risk you need monitoring at an enhanced level. If you choose not to take it there’s the risk that when you are owned, fined and censured that there would be literally nowhere to hide from a culpability perspective and business owners will not want that reality. 

Bearing in mind this includes key material appliances for above and below classified spaces in air gapped and non air gapped appliances, appliances on trading room floors and banking environments then you start to see the issue amplify. A high percentage of this kit will never receive a patch for this problem.

What does the industry need to do ? The reality is it knows exactly what it needs to do and that is to be better community bedfellows, partake and contribute back and also be better open source citizens and think about how you develop, release and support. 

This Intel issue, however badly managed it was, may just be a klaxon call. Lets hope so.

New beginnings


Since I decided to leave my security role at Red Hat last October 2016, and finally exiting a year ago today in December I’ve been somewhat busy. I actually planned to be less busy and to just do part time consulting and writing for a couple of years but those plans went out the window at haste.

In January I started a consulting company AuditSec Services with Chloe my wife to do virtual CISO roles, writing and speaking and to collect software royalties and fees in a tax efficient manner on my terms. E.g not to be working 100+ hour weeks that had become the norm at Red Hat, and to see more of the kids and genuinely enjoy more family time.

Well. AuditSec went nuclear. By the end of March I’d already been in six countries doing consulting, public speaking and providing security guidance to companies in four verticals at C level. I’d been on stage in countries as diverse as Holland and Slovenia, Germany and Russia and I was more busy than I had ever been.

If you’re going to do your homework, get paid for doing it

Then a stint on contract at Gartner helping them define their security practice as a Director and as their acting CTO for their global security practice whilst also fulfilling my role as a Director of the Cloud Security Alliance now that I wasn’t handcuffed by the dumbest employment contract ever at Red Hat.

Then life changed somewhat. Prior to meeting Gartner I met with a company in Birmingham, Falanx Group, a security and intelligence house that had been through a rebirth and were in a period of transition. Exploring how to be an MSSP of sorts, with their own SOC and the usual blend of security services they were intriguing in many ways although I had some reservations that needed to be put to bed. At Gartner I worked with Secureworks, Tata, Wipro, BT, BAe, and products such as Splunk, Carbon Black, Vectra and CrowdStrike on seven figure projects with FTSE50 companies RFP processes and the entire period was spent doing the most dynamic and intelligent market research and strengths and weakness analysis of the MSSP market that anyone could ever do. And to be paid for doing it.

A somewhat unique experience, to be sat wearing the senior Gartner hat at the table. Actual research I could use to shape what I would be doing for the next stage of life. Research not just into capabilities but weaknesses in core service delivery and also commercial weaknesses that blindside almost every MSSP SOC offering other than Secureworks. Add into that the fact that the whole service proposition is plagued by the fact that margin maintenance for providers has never been harder as CrowdStrike and Carbon Black’s investors both realise that they have to have a service play which then throws contractual agreements with Verizon / BT / BAe / Wipro / Tata (add MSSP player of your choice here) into a position where pre IPO or pre exit there isn’t the appetite to maintain product roadmap in search of value added services – that compete with their partners.

SOC providers are servicing customers with 2015’s enterprise needs

Be under no illusion every SOC provider is going to spend 2018 trying to sign customers to 3-4 year deals ahead of time to maintain profitability. If they don’t then Splunk (whose share price has underperformed dramatically since launch in a bull market for governance and compliance) and others are going to see revenues continue to drop whilst differentiators in the marketplace bring services to the fore. Elastic’s scheduled IPO is going to be amazing, Josh Bressers my number two at Red Hat and host of the Open Source Security Podcast, (who left Red Hat with me is now heading up security there after I turned down the role and suggested they go hire him) is now at Elastic. Even SecureWorks whose share performance has failed to please analysts has begun to rethink how it re-examines it’s customer relationship away from the backward steps it took whilst part of Dell/EMC.

Amazon’s launch of their second generation monitoring capabilities at ReInvent this week means that the intelligent design of solutions that meet the needs of the DevSecOps generation is close. The smart money is not on traditional SOC, it’s how you bridge the gap, the gap I’ve described in presentations on stage as way back as CSA EMEA Congress in Spain last year whilst still at Red Hat. You can find my slides from that deck online (click here for a PDF copy of the deck) and also repeated in a talk I gave in my other role as roaming techhead at BrightTalk (go be my judge and listen for yourself clicking here to register and listen to our last session).

Now at the time in Madrid in October 2016, there were a lot of very qualified suited and booted folk in a packed auditorium, a room populated by C level decision makers from across Europe. I know talking to some afterwards they felt that I’d punched them violently by describing the problem and their failures to use technology and people to solve it. In fact during delivery of the deck I could hear the audience discussing it with almost audible winces. Never a good sign when you’re only armed with a laser pointer. However, the team from Google who were in the audience went away and used my deck to re-address how they talk security across their business. In Slovenia in April this year at another event I was keynoting with Red Hat and the Slovenian Government they came back to me and thanked me on stage for creating them a “pivot” point in their monitoring and release capabilities. I knew I was starting to be heard by audiences and being influential in decision making about rethinking process, event, monitoring and compliance even if folk felt wounded in the room each time I delivered the deck.

But then when the biggest bank in Spain who had been in the audience for my talk also adopted the same mindset, that was the gamechanger. I knew then that change was necessary and traditional SOC and SIEM solutions just service latent legacy needs. Even Red Hat were getting it badly wrong and only thinking about OpenSCAP and CVE compliance as their take on security which was one of the reasons both Josh and I left Red Hat’s leadership team.

MSSPs and SOC providers haven’t evolved fast enough, or can’t without adding more technology (and latency and security risk) to address the pressures of where companies development mindsets and pressures are forcing them to operate. This is especially critical around a multi cloud estate or where organisations are starting to use mobile development teams to garner new revenue opportunities. Flexibility of approach and a genuine need to talk the same language as your customer is at the core of a provider mentality and sadly the industry is about margins and the provisioning of hardware from competing manufacturers to deliver their concept of a value add. The customer doesn’t really stand a chance and no matter the RFI/RFP process usually buys 60% of what they need and 40% of what they’re given contractually and then have to mitigate against. In whose eyes is that adoptive agile security for todays business needs ?

If you read my regular articles on TheStack (my security articles in 2017 are the most shared real estate on TheStack), especially where we touch on process failures then the need becomes very real. And until now I’ve not had the tools to match my doctrine. Until now.

An overlooked fact

The SOC/SIEM marketplace is segmented and populated in a mixed hotch potch of stacks of technology from pre IPO companies under the cosh and unable to determine roadmaps and pricing, harnessed by market leaders unable to influence that roadmap or pricing but reliant on core capability. When you don’t know what your customer wants and you turn up with an Ferrari (which won’t solve their problems) and your customer has a problem you don’t understand and only has Skoda money then can you shift your business to the left and take a haircut enough to deliver capability ? Not if you’re reliant on the current model of stacked products that only give you 65% of what you need with your large customers.

Do Enterprises have the money to deploy what they think a SOC is ?

The SOC and SIEM market is one that is overpriced, misunderstood and badly designed even if you are an enterprise throwing £3m a year at it, what you get is more a bet than a technology partnership. Sadly where the SIEM industry is at is delivering a chocolate box approach to security event reporting, intermediate barely useful threat reporting without actually asking the industry what it wants. We see it year after year at RSA and if you listen to any of the podcasts I recorded from the floor of the Moscone at RSA in San Francisco the penny will start to drop.

Anybody remember The Likely Lads ?

During the period March to August 2017 John Blamire the founder of Falanx, a lovely guy who hails from the North East of Britain like me, messaged me almost daily. He talked to me at length about coming in to help them through a transition and to provide a much needed shot of inspiration into the company, below the radar. John knew what I had done at SmoothWall and Zimbra and the work I’ve done with Red Hat and the podcasts and at CISO level in the UK, EMEA and US. There was a job to do at Falanx and the role wasn’t going to be either easy or without significant challenge but that would require inside knowledge at the highest level of the MSSP market. The significant research I’d been able to do at Gartner was going to provide us a core level of understanding towards feature development and reporting needs in a UK market dominated by “GDPR experts” who aren’t listening to the wider needs of businesses.

The acquisition of AuditSec and next steps

In September 2017 AuditSec Services was acquired by Falanx Group for stock and I became Group CTO. The below the radar transformation and rebuilding of Falanx was to begin. Now the hard work would begin. The important piece of that paragraph is the line “for stock”, not cash, not cash and stock, stock. It meant skin in the game.

Commercially I have been successful enough to be solvent and comfortable in life and I’d need to be if I was going to take a fifty percent plus haircut on salary with a small company like Falanx unable to pay me a solitary employment benefit compared to say Gartner. No shiny contributory pension, no car allowance, no family health benefits (I have a wife and two children 6 and 4) only the promise of long hours, indigestion and long trips up the M5 from Bath all hours of the day and night. I don’t need to be at Falanx, I’m choosing to be there as long as I’m heard and people follow my lead to drive revenue and product. Six months of John wearing me down to join made me realise that for the first time since SmoothWall in 2001 I had an opportunity to make a commercial gain of a significant nature, even with a smaller holding. But matched to an aligned promise to come in and be heard and have a business that I could shape. So for FLX shareholders thinking that expenses or expenditure is an incentive, not with Mike Reid on watch. My bank manager must think I’m slightly bonkers, thankfully my wife is supportive and sees the bigger picture.

Since then business process re-engineering has been my focus and looking at every aspect of the business and the creation of a suite of tools and technologies that we are about to go to the market with, that nobody has seen coming. I am very proud of what we’ve been doing under radio silence, aided by partners and longterm friends providing input from my Open Source Rolodex.

This is personal – taking the principal steps needed to drive demand and growth

In 2001 I launched SmoothWall commercially. I took a back bedroom company to a reseller market in over a dozen countries and turned it into a company with products, services and revenue that didn’t just compete but became the standard.

What we have been doing very quietly the last few months at Falanx is based on the same principles, the same ethos but with better developers and more capabilities to bring to the fore. Only what we’ve had to do at Falanx is to do it silently under the radar without market awareness, without analyst focus and to build something that has demand and meets my security criteria.

What else can I tell you ? – Not a lot, but you know I wouldn’t be involved if this wasn’t about to make people sit up. The established firewall and filtering market had to listen to me when I stood up SmoothWall. The MSSP market – it’s time for a shake up and this time it’s using qualified research looking at where the weaknesses in the most expensive platforms can’t play well.

I sit and read the Falanx investor boards once every couple of weeks and laugh nervously. Everyones an expert. Well newsflash. I’ve been here ten busy weeks and I’ve not seen daylight. Not one of these posters clocked my arrival. The fact the security company I created on thin air just got bought out in an MBO backed by Tenzing afer becoming the defacto standard.  Too funny, I’d have really thought someone would have said “theres this guy from Gartner who just landed, who sort of has a track record in making revenue and marketshare – why did he get in at the bottom ? Why did he choose Falanx all of a sudden, what does he know that we don’t ??”. Nothing. Radio silence, not one blink. It seems that folk are too busy speculating about stuff to spend time doing due diligence which is hilarious for me as a shareholder getting in at a stupidly low market valuation.

NOBODY has a clue what we’ve been doing, what I’ve been driving, and this is going to be if nothing else a positive shot in the arm and to make people sit up fast.

I don’t put my name on things that fail, and theres a reason I’ve been very very quiet. I haven’t even appeared on the Falanx website out of choice although that will change any day soon.

However folks, when we go live this is going to be done properly with a service that people consume because it’s world class. I know one model, make longterm growth revenue. Following the SAME model as I created at SmoothWall at a price break where the competition don’t have an offering to compete in a market that has a need for better security capability. Cloud is here, enterprise workloads are migrating, security compliance is harder and if you go read the articles I’ve written in the press over the years (see the press section in the site menu) and read between the lines then you are armed to make a sensible guess about whats about to land.

Watch this space, subscribe to my podcasts along with the tens of thousands of security folk, analysts and journalists who choose to trust what I have had to say for the last five years to the intelligent masses. If the CIO of US.Gov listens in maybe you should too.