Every Client Encrypted in 90 Minutes: How This MSP Survived

Robert Cioffi (0:06): And it lives with you, it stays with you. Even though it's five years later, I still I still, you know, I get that anxious feeling, but this is a coping mechanism for me. It's cathartic to be able to talk about it. And if I get one person that reaches out to me and said, you know what, I'm taking some additional steps to better my company, that's well worth what happened to me.

Andrew Moore (0:28): Alright, welcome to the How to MSP podcast. My name is Andrew Moore. I am your host from RidgeView Advisors, and I am joined today with a very special guest that I'm super excited about. Robert Cioffi with Progressive Computing. He is the CTO and co-founder of the organization. They are an MSP. And

Robert Cioffi (0:47): Yes. Yeah.

Andrew Moore (0:50): he's got a remarkably interesting story that I am really excited about. Becky Teal from Huntress introduced us, and I know.

Andrew Moore (0:57): From talking to folks, he's in the channel doing a lot of things with different organizations, not just his MSP. So, Robert, thank you for for joining us today. I appreciate you being on the

Robert Cioffi (1:07): It's my honor and privilege to be here, Andrew, and anything I could do to get the story spread out a little wider into the community the better. so I just appreciate the opportunity to speak to your audience.

Andrew Moore (1:21): Awesome, awesome. so a couple of questions. First, we'll start with and I'll let you go into all of it. where are you at physically? Where does the podcast find you today? And then tell us a little bit about who you are and what you do and kind of why people should should care about your opinions and thoughts on things and in the channel. What have you been up to? Let's talk about that.

Robert Cioffi (1:40): They should care about my opinions because I have them.

Andrew Moore (1:44): Welcome to social

Andrew Moore (1:46): media, right?

Robert Cioffi (1:46): How yeah, how is that for entitlement? no, I

Robert Cioffi (1:49): mean, you know, I think I have a a a pretty good story to tell. it's a pretty awful story, but we'll get into those details later. But a little bit about me and us. progressive computing was a company that I co-founded in nineteen ninety-three with my current business partner. We're fifty-fifty owners. we started out way back in the long ago before time in nineteen ninety-three of doing application development. We were building

Robert Cioffi (2:16): character-based DOS software. I mean, this stuff wasn't even networked. It was these were independent D-base type databases. if

Andrew Moore (2:24): Wow.

Robert Cioffi (2:24): anybody remembers Clipper, reach out to me. I bet you got a lot of gray in your hair like I do. but our business morphed over those many years and it it follows the the typical trajectory of what IT

Robert Cioffi (2:38): looked like we got into networking, NT came at you know, Novell, NT, then cloud, cybersecurity. Somewhere around 2008, we were a little bit early on the MSP transformation. but we're a traditional MSP based out of Yonkers, New York. if you know your geography, that is just north of New York City. We border the Bronx. Bronx borders about maybe five miles that way.

Robert Cioffi (3:05): in case you're wondering where it is. but we were you know it's a city of two hundred thousand people, but I have probably twenty-five million neighbors in the you know fifty mile radius. yeah, and just you know, got very involved in the IT MSP community decade and a half ago, got very involved in peer groups. I facilitate two beer gr peer groups as an IT Nation Evolve owner peer group.

Robert Cioffi (3:34): facilitator. and just just you know, I love the community and community is a word that I'll repeat often as I tell my story about a huge success factor in us navigating through what essentially should have been a business ending event.

Andrew Moore (3:54): And and one of the things that we talked about briefly before we move on is you had mentioned there's this flag for those who are not watching and that are maybe listening to us. So there's this really beautiful American flag back behind you in your officer. Can you give everybody a quick background on that? 'Cause I know that there was a story there.

Robert Cioffi (4:10): Yeah.

Robert Cioffi (4:12): well, it's a it's a flag that I've had in my possession for decades. I mean, since I was a teenager, it it it came into my possession and I've never known what to do with it. I've you know, obviously other than hanging it outside a home, but I always felt like I don't want it to get weathered, you know, I wanted it to be someplace

Robert Cioffi (4:33): that I could sort of display it. And then once COVID hit, behind me are these blinds and shades and everything went video. So I needed something to cut back on the glare. So I was like, all right, let me put this up as a temporary. And it it stuck. the number of calls that started out ten minutes late because people wanted to know or comment about the flag was a very big surprise to me. So you know my parents are off the boat from Italy.

Robert Cioffi (5:01): They came to this country with nothing, literally quite nothing in their pockets, other than a trunk full of clothes. And they put four sons through college and my three brothers are all extremely successful. I'd like to think that I've foll followed in their footsteps a bit. but it's you know, I'm just a very patriotic individual who loves this country and it just for me felt like the natural easy thing to do to put up an American flag behind me.

Andrew Moore (5:30): I think that's amazing and I would I love that I love that there is still a level of true patriotism that runs through the the country in a way that really describes what this country is about and that's giving people opportunity. Right.

Robert Cioffi (5:45): I think there's mo I don't wanna get political here, but I think there's a lot

Robert Cioffi (5:48): more of that than media would suggest is present. I think it's a lot more prevalent than what we're being told.

Andrew Moore (5:57): I I I would agree with that too. And I think that there are a lot of people who are proud of this country and sometimes that maybe don't feel like they can can share that. And I love it when

Robert Cioffi (6:04): Yeah.

Andrew Moore (6:04): when people do feel like they can they can and they have a great story behind it. So I applaud that. I appreciate that very much. so let's get into this a little bit. So

Robert Cioffi (6:12): Yeah.

Andrew Moore (6:13): Becky and I were talking at the end of the podcast, she said, You want you to talk to Robert, he's got a very interesting story. And she's I'm not gonna tell you what the story is. She's

Robert Cioffi (6:21): Mm-hmm.

Andrew Moore (6:21): like, but you guys should have a conversation. And so I reached out to you and then we we started talking and

Andrew Moore (6:26): You

Andrew Moore (6:27): told me about your experience, and I was absolutely entranced because of my situation. I think one of the things that we saw, and I'll preface this a little bit, when we had RMSP, especially in the post-pandemic era, right? Probably starting around 2020 and past that, with a number of cyber incidents that were not only happening to our clients, but we were seeing happen in the

Andrew Moore (6:56): Industry in total, there was a real concern around our my my COO's group and other operators that we knew. It was like we are one event away potentially from taking all this sweat equity and all of this work and all the things that we did from having a worthless business and a bad reputation as operators because we couldn't protect our clients, and so it was a terrifying thing that would really drive a lot of the decisions that we made, you know, and how we

Andrew Moore (7:26): built security and what we tried to do to protect our clients and ourselves, but it was always lingering in the back of our head and it gave us it gave me a lot of anxiety. And so

Robert Cioffi (7:35): Yeah.

Andrew Moore (7:35): I've never met someone at this level that had experienced something like that and come out of the other side of it in any capacity. And so when you began to tell me your story, I was fascinated by it. So could you kind of start from the beginning and explain to our listeners what happened to you and where

Robert Cioffi (7:50): Yeah.

Andrew Moore (7:51): where you're coming from and and what what brought us to this conversation today?

Robert Cioffi (7:56): sure.

Robert Cioffi (7:57): unfortunately we don't have hours to get through all of the details. So you and the audience will forgive me if I'm you know, leaving some cliffhangers or not painting a full picture. I've learned that I need to kind of navigate through this very quickly. brevity is not my forte, Andrew. So I I tend to be a little I tend to get deep into the story sometimes.

Robert Cioffi (8:22): But essentially what has happened to us is exactly the nightmare scenario scenario that you painted, right? A lot of MSPs will have anxiety or have anxiety over this notion that, hey, one day a threat actor could potentially own all of my customers simultaneously. and you know that would be, you know, a really catastrophic, terrible thing to happen. let me let me start with

Robert Cioffi (8:51): Invoking the terror and anxiety that you feel when one of your customers becomes ransomware. Right? Every MSP that I know that I've ever spoken to can raise their hand to say, Yep, that happened to me. And man, we sweat bullets. I lost sleep. I was maybe even in the bathroom puking for a few minutes, you know, just thinking about like how awful this is. the the

Robert Cioffi (9:19): the stress level, the the anger, the the blaming, the the long very hours over many, many days to put Humpty Dumpty back together again, right? Now take that feeling and multiply it not just by the number of customers that you have in total, but maybe maybe put an exponent on that as well. Because what happened to us

Robert Cioffi (9:47): was on July second of twenty twenty one, a threat actor had gained access to our RMM tool, Caseya VSA, and they broke through a zero-day exploit, uploaded Reval Sodno Kibi ransomware in a little package called agent.exe, and then instructed every device under control and management within that RMM tool to install that program. And since

Robert Cioffi (10:15): most computers had an SSD drive those in you know five years ago and were you know good performing machines. That ransomware took about 90 minutes to install on every single endpoint that we managed, which totaled about 2,500

Andrew Moore (10:30): How many? How many endpoints?

Robert Cioffi (10:33): endpoints, 250 of which were servers. This was 80 clients in 200 physical locations in four time zones, right? so that was

Robert Cioffi (10:44): initially what had happened, right? So we're sitting there, we're getting reports of something happening, a lot of confusion. That's the story in terms of the metrics and the numbers of exactly what happened and the the im the numerical impact.

Andrew Moore (11:02): What so when the

Andrew Moore (11:03): agent was deployed,

Robert Cioffi (11:05): Yeah.

Andrew Moore (11:05): was it readily was it apparent that the payload had actually done something? Or was it just like, we know that this has been deployed and we're waiting for what the next shoe to drop? Like how did that happen for you?

Robert Cioffi (11:15): No, what

Robert Cioffi (11:16): happened was and I can tell you like, you know, where I was and you know, exactly how I found out, but we had essentially found out after the damage was done, right? Because as ransomware is encrypting a machine, you don't know that it's doing it, right? It's not gonna you know, raise its hand and announce, hey, I'm starting to encrypt your machine. it's gonna do so quietly behind the scenes until it is complete.

Robert Cioffi (11:42): So the reports that we were getting in the early afternoon of Friday, July second, twenty twenty one, were reports of things, you know, things not working, right? Or systems being inaccessible. and that was the very first telltale signs that, you know, something widespread was going on, ex especially since our phone was ringing off the hook, coming in from just about every one of our customers.

Robert Cioffi (12:13): and that's I think, you know, where the emotion if I if I can paint that picture for you, I was sitting in my kitchen at about noon well it was about noon that day. It was remember this is Friday, July second, right? Right before, you know, talking about the American flag here, right before Fourth of July day weekend. and where where are most people's heads on a day like that, right?

Robert Cioffi (12:38): they're probably thinking, hey, it's about noon, I'm checking out for the day, I'm taking off early. The weather here in New York was spectacular. you know, we can't always say that here in the Northeast. so everybody was kind of checking out for the day and I was doing something very unusual at that hour. I was eating lunch. I know that's crazy to think that a business owner and a or an MSP would eat lunch at noon, but

Robert Cioffi (13:05): There I was in my kitchen. It was about twelve, maybe twenty, twenty-five at that point. I was chatting with somebody, j you know, joking around, just like kind of thinking, hey man, we're done for the day. Ticket counts are dropping. and I saw my director of operations come up a set of stairs. I sit on the second floor of the building. He came up the stairs, was walking down the hall, and I could just tell looking at him that something was

Robert Cioffi (13:33): was really, really wrong. in fact, when I looked at him, I thought he looked like somebody who was coming to deliver some really terrible news. in fact I had commented to somebody who was in the kitchen with me and I said the words, Holy shit, I think somebody just died looking at Jay, you know, coming down the hall. So I wasn't

Andrew Moore (13:55): Yeah.

Robert Cioffi (13:55): sure whether I was getting up to console him or whether I was bracing for impact that he had something terrible to tell me.

Robert Cioffi (14:03): It was the latter, and thankfully nobody did die. but it certainly felt like I died that day. so I met Jay right outside the kitchen and he was pal you know, he's Filipino, right? So he's got a dark complexion naturally, but he was white as a ghost. He was sweating, and we have a great professional relationship, but he would not look me in the eye. so I knew that I had to like like

Robert Cioffi (14:29): pull it out of him like I knew something really terrible was happening and I'm like, Jay, like what's wrong? And he was just pursed lips, shaking his head, and he looks up at me and he said, every one of our clients is ransomware.

Robert Cioffi (14:46): And I apologize for those that are listening. you might hear me pause and it's because I do need to choke back some of the emotions sometimes as I relive through this. But it was it was completely surreal, Andrew, for me to hear these words like like it was absurd. Like, what do you mean all of our customers are ransomware? Right? And in fact that is exactly the

Robert Cioffi (15:12): distaste that I had in my mind. Like what like what is this garbage you're handing

Andrew Moore (15:15): Right.

Robert Cioffi (15:16): me? Like that's not a reality. What do you mean? Like I threw it back to him and I said, Jay, what do you mean by that? Like, like which one? And he goes, No, Robert, it's all of them. And still I tried to reject his reality and insert my own. at which point he was insisting that it's you know that it was all of them

Robert Cioffi (15:37): Now I'm gonna pause in the story here for a second because something happened in a split second or two or five. We're now standing in a hallway on in the middle of our office, and there are cubicles and other offices sort of around us. And now I can hear and even see what's happening with other people. And I'm hearing and these are snippets of conversations. so here we go. I don't know what's happening.

Robert Cioffi (16:04): please calm down. I think something system wide's going on. hold on, let me see if I can get a technician. I could see people standing, you know, from their seats on the phone now. I can see some confused looks. I can almost feel their their discomfort of in the conversation they're in. And it's and it's a hundred percent. I mean, I I had my eyes and ears on about five or six different people.

Robert Cioffi (16:34): And that was what they were they were all on the phone saying something similar to that regard, right? Trying to calm people down. I don't know. Let me get some answers. Let me get back let me get right back to you. so at that point I kind of knew like, okay, maybe there's some truth to what Jay is telling me here. And I said, I pushed on him again. I said, Jay, okay, you gotta tell me which ones.

Robert Cioffi (17:03): Now he did something that I don't think we have a little bit of a debate whether or not he did this purposely or he did it subconsciously. but he started to list customer names after I insisted on him telling me. And it's a little bit of a rhetorical conver a little bit of a rhetorical question here since the audience can't respond, but I want you to think about what order he might have given me that client list in.

Robert Cioffi (17:31): All right, Andrew, you care k care to take a guess?

Andrew Moore (17:34): My guess would be so there would be two different ways I would I would probably start with our biggest accounts. I would I would start

Robert Cioffi (17:39): Yeah.

Andrew Moore (17:39): with the ones that were I'm like it would be the it would be the fuck factor. It would be like, you okay, you don't believe me? Like let me talk to you about the the nasty ones, right? I wouldn't start with the small ones. I'd be like it's you know this one, this one, this one. I mean just for dramatic effect. Like I would just be like this is where we're at.

Robert Cioffi (17:54): It was

Robert Cioffi (17:56): it was essentially I'll say revenue order. He got to about

Andrew Moore (18:01): Yeah.

Robert Cioffi (18:01): number ten and I begged him to stop. each one of them felt like I was like each name felt like it was landing on me like a slab of stone and pancaking me into the ground as I heard each name. I just sank lower and lower and lower. I spoke briefly.

Andrew Moore (18:20): How did you how

Andrew Moore (18:21): did you not just like like I would have literally walked over to the trash can and thrown up? And then I probably would have stood there for a second and been like, is it is it okay for me to just leave? Like I like that would have been that fight or flight thing that would have kicked in and it just would have been like, I I have to I have to walk away from here for a second. Like I what how did you guys just

Andrew Moore (18:46): Did just like throw cold water on each other, slap each other in the face? Like did you just go into a room by yourself and scream? Like what where what how did you guys move in move past fog into action? How did that work?

Robert Cioffi (18:55): sort of all of the above. Yeah.

Robert Cioffi (19:01): well, I've been in a I've been in a serious car accident once. and if you've experienced any sort of physical trauma in your life, there is a there is that shock value, right?

Andrew Moore (19:14): Right.

Robert Cioffi (19:15): And so here I was, now my brain felt like it was expanding a hundred times its size, trying to push out of my skull.

Robert Cioffi (19:23): because of all the thoughts swarming in it, right? And the emotions and logic that were fighting each other about what to do, what to think, what what you know, do I crawl into a hole? Do I do I just leave for the day and pretend nothing happened? I mean, those are thoughts that kind of went through my head. I'm not gonna lie about that, right? Like, do I just abandon ship? Like, 'cause the thing's on fire. some of those thoughts came a little bit later.

Robert Cioffi (19:49): But my initial reaction was I did go into this sort of command and control mode of, okay, we we've we've got to assess what the hell is really happening here. Cause I'm not sure if the words that I'm hearing are true to reality. And

Andrew Moore (20:05): Right.

Robert Cioffi (20:06): I think that a little bit of denial was was was prevalent there, right? I was a little bit in denial still about what I was hearing, even though it seemed to be validated by

Robert Cioffi (20:17): converse snippets of conversations that I'm either witnessing or hearing in the office. So I grabbed Jay and we came exactly where I'm sitting right now with exactly what you see in my office here if you're if you're watching. with the same exact three screens that I have right here right now. might have even been the same computer, who knows. and so I just started going into like go mode in terms of a technician. Like I

Robert Cioffi (20:46): I need information. I need I need data. I need I need to see things for myself. I don't believe what I'm hearing. I need validation with data. and so at this point now, Jay is standing like literally right over my left shoulder, and my phone is ringing, my cell phone is going off, Teams is blowing up, my text messages are blowing up. I've got now my two chief technical people.

Robert Cioffi (21:13): on our team. At this bus at this point of our business, we're in business 28 years at this point. And Mark and Scott had been with us at the time, and they're still with us today, I'm proud to say, at the time, they were with us for 23 years. So these are guys that I like, they are the fabric of our DNA. Trust them implicitly, explicitly. so I had one on speaker phone and one on you know my cell phone and we're trying to like dig through this.

Robert Cioffi (21:43): and that's when I had a complete validation of of the ransomware incident when I'm sitting at my desktop and I can see all the icons on my primary screen start to flip from their typical Word, Excel, or other familiar icons, and they turn into white boxes. So

Andrew Moore (22:07): So you got cryptoed with your clients.

Robert Cioffi (22:10): Correct. Except for one particular file, right? The readme dot txt file. that was unencrypted. and of course I opened that up immediately and knew that we were we were toast, right? So of course because we had used the you know, we we eat our own tacos here, right? The same tools that we deploy to our customers is what we use to manage our own systems.

Robert Cioffi (22:38): We can debate whether that's a good practice or not. But remember, this is five years ago and

Andrew Moore (22:45): Right.

Robert Cioffi (22:45): you know, most, you know, there there there are things that we've learned all learned from those days that have changed our perspective on things. but that's the moment, Andrew, that you were starting to describe earlier, where I put my head in my hands and I I froze. I didn't I didn't know what to do.

Robert Cioffi (23:08): There was a good five or ten minute period of time where I had contemplated things like, do I just go home? Do I power off my cell cell phone? Do I dismiss everybody? Am I gonna need to fire everybody? Right? what do I say to my clients? Right? So let me put it this way. As a as an MSP owner, most of us started this business as technicians, just like I did, right?

Robert Cioffi (23:37): and so I'm gonna write a business book one day on business type personalities and it's only gonna have one personality. I know you know these other systems and all these other personality types. My business book only has one personality type, and it's called Fixer, right? And it's only a one page business book because in a paragraph or two I can tell you what that personality looks like. There's a problem, I can fix it. and I'm not talking about just IT, like

Robert Cioffi (24:04): All the stuff, all the trades in my house, I self-learned, I do them all. I can fix basic things on a car. that's I think that's also what made me, I'll I'll say, a pretty good business person is because I just

Andrew Moore (24:18): Right.

Robert Cioffi (24:18): look at business problems as a problem to fix, right? Just apply logic and reasoning and a little bit of that grit, right? Of just being determined to be able to fix things. The fixer was broken here, Andrew. I had no idea.

Robert Cioffi (24:33): What the hell to do next? I didn't know who to call, what to say, what to think, what to feel. I was staring into the abyss, contemplating my own mortality. I watched 28 years worth of sweat equity, blood, sweat, tears, and toil evaporate before my eyes. Right. We had a multi million dollar valued business whose valuation not only just went to zero.

Robert Cioffi (25:03): But might have even gone gone negative, thinking about all of the lawsuits that might be coming our way. I thought about recalling my daughter home from college that she would not be able to finish because I wouldn't have the funds to pay for that. I might lose my house. I would have to fire all of my employees. I mean, these were the dark thoughts that started to swarm in my head altogether. I mean, any one of those demons is hard enough to battle, but

Robert Cioffi (25:29): 20 of them showed up at my doorstep all simultaneously. so it it took a while for me to snap out of it, and thankfully Jay was with me here because he did and said something incredibly I'll say it was silly, right? but it was exactly the silly words that I need. He saw me faltering. He saw the guy that.

Robert Cioffi (25:57): like put so much effort into building this organization and was the go-to answer guy for everything. And he just simply put his hand on my shoulder and said, Come on, Robert, we got this. And my first reaction was, What the fuck do you mean we got this? Right? Like,

Andrew Moore (26:14): Eight.

Robert Cioffi (26:16): are you fucking kidding me? Like we got what? Like we just got screwed, like to the nth degree here. No, we don't have this. Now this is what I was thinking on the inside, right?

Robert Cioffi (26:26): But I looked at him and said, son of a bitch, he's right. Like we do have

Andrew Moore (26:29): Yeah.

Robert Cioffi (26:30): this. I have no idea how the hell we're gonna fix this, but we're gonna fix it. So let's go back to what you said earlier about fight or flight. I thought about flight, but I chose to fight. And that's

Andrew Moore (26:41): Yeah.

Robert Cioffi (26:42): w that's what's next.

Andrew Moore (26:44): I love that because I used to talk to my the owner of our company whenever things would get tough or whatever and it just I I'm a big pop media guy too. I like to watch shows and movies and stuff and I'm a fan of Star Trek and there was this Picard quote that I really liked, which is one impossible thing at a time.

Andrew Moore (27:03): Right? One impossible thing at a time. And so when we

Robert Cioffi (27:04): Yes. Yeah.

Andrew Moore (27:06): would sit there and be like, my God, there's like all these things that are in front of us. I would sit down and I'd be like, okay, dude, one impossible thing at a time. What are we going focus on first? Right. So as as I'm very kindred spirit with you in being a fixer. Like I am very, very, very similar into your personality. I like to know how everything works and take things apart and put them back together and do my own home improvement projects and all those things. So I'd love that that's where your mind went, which was, okay, we're gonna fix this.

Andrew Moore (27:33): So what did you do? What was step one? Then what would like

Robert Cioffi (27:37): Yeah.

Andrew Moore (27:37): how did how did you start putting one foot in front of the other? Like how did you come out of the fog and how did you start getting people involved? And like what what led you out, you know, both short term and long term to start, you know, rebuilding the business from that day?

Robert Cioffi (27:51): well I'm

Robert Cioffi (27:51): a huge Treky too, so we can geek out that on that some other time.

Andrew Moore (27:54): Nice.

Robert Cioffi (27:54): but I'm also a big history buff, so I started to think about some inspiration from historical figures and I thought about a quote from Winston Churchill during World War II in the face of annihilation, and he famously quipped, When you're going through hell, keep going. so I felt like I was standing at the the gates of hell, right? If I can quote Dante, right? abandon all hope, ye who enter here.

Robert Cioffi (28:20): Or in an you know, there's the Italian version of it, which I'll skip for now. but that's that's where I felt like I I can't go backwards, sideways, I can't stay where I'm sitting. I gotta push those gates of hell open and start walking through. so just sort of accelerating, we gathered the whole team into the conference room. And as people were filing in, my mind was still a blank, Andrew. I'm like, what?

Robert Cioffi (28:49): Am I going to say to these people? What am I going to tell my team members? People who I would consider to be my friends that we've worked together for so long. And then even beyond that, what the hell am I going to say to my customers? and it was interesting because at that point we were about eight years into running EOS here. So for those who don't know what EOS is, just a system for running your business, but one of the

Robert Cioffi (29:18): components of EOS is to define what your core values are. so our core values were posted up in our conference room and by some miracle, by some grace of God, I looked up at them and said, that's the answer right there. it is our culture. It is the like is it, I believe it's Peter Drucker who said that culture eats strategy for breakfast, right? I had

Andrew Moore (29:42): Hm, yeah.

Robert Cioffi (29:43): no tactical plan.

Robert Cioffi (29:45): I had no strategic vision for how I was going to come over this, but I knew that I had an amazing culture here. And when I say I, what I'm really mean is I'm misspeaking here. We had an amazing culture here. So our four core values are, you know, team together, we get it done. commitment, determined to do what it takes, respectful always and humble confidence service without ego.

Robert Cioffi (30:15): So I knew that those four core values were gonna be paramount to our survival. And so I started off talking to the team about look, you know, acknowledging what's happening here, this is really awful, but I want you to think about those core values repeatedly and keep them in your head in every conversation, in every email, in every text message, every ticket, every site visit between now and

Robert Cioffi (30:43): And the next I don't know how long, but this is what's gonna get us through. So while that alone wasn't enough, it was enough for that moment to be able to pull the team together and snap them out of that, you know, tremendous feeling of dread and woe that we were all collectively feeling. I needed to give them some sort of hope and semblance that, you know, one of the two leaders here, the top, you know, two people.

Robert Cioffi (31:13): had the the strength and confidence to instill in them a sense of we've got this, right? Just as Jay said to me. and but here's the reason why we have this. We are not going to fail because of who we are.

Andrew Moore (31:27): Yeah, and then as you started to move through that process of getting your team back and re-engaged, just real quickly, like how did you understand this is the vulnerability? I mean, because we use so many tools.

Robert Cioffi (31:37): Yeah.

Andrew Moore (31:37): Like for all you knew, it could have been the antivirus solution, it could have been anything, it could have been somebody that hacked your environment, like through it could have been one of your former employees. Like at that point you didn't know what you know today. Where did you go forensically? Like how did you start to really say, Well now I need to turn this on and then this is what I need to do, and then

Andrew Moore (31:56): How did that lead to them figuring out what happened, how to remediate it, and then how you guys were able to start rebuilding? Like what did that look like?

Robert Cioffi (32:06): Yeah, so for the next forty eight hours, right? Remember this is a Friday afternoon, July f Monday, July fifth was you know, national holiday. So we had that three day quote unquote weekend. yeah.

Andrew Moore (32:17): Can let me let me put a pin in that for one second, keep

Andrew Moore (32:19): your thought there, because I wanna I wanted to mention this earlier. For those who are not familiar with how these types of events work, usually they're before some sort of like holiday or an internal

Robert Cioffi (32:30): yeah.

Andrew Moore (32:31): event.

Andrew Moore (32:32): That the hackers are aware is going to be potentially disruptive to your business. So a lot of the times you'll see these sort of things happen like right before Christmas or right before Thanksgiving

Robert Cioffi (32:40): Yeah.

Andrew Moore (32:40): or right before July 4th or something because they know you're gonna be short-staffed, you're not prepared for it, right? Like you're you're you're probably taking it easy. and so I just I want to make sure that people really get a feel for like that. This wasn't like just a randomly chosen day or that they were trying to make a point because it was the nation's birthday. It was because this was a strategic advantage for a hacker, right? That's what they

Robert Cioffi (33:01): Hundred percent.

Andrew Moore (33:02): Yeah.

Robert Cioffi (33:04): a hundred percent on that. so and again, there's a lot of details here that I have to kind of just step through quickly, but

Andrew Moore (33:12): Right.

Robert Cioffi (33:12): we had notified our cyber liability insurance company, which by the way, our policy went into effect that Monday and this was a Friday.

Robert Cioffi (33:21): Well, we had a policy prior to that, but it was a garbage policy. So we had now a policy in force. I still would urge everybody to read their policy in clear details and maybe get some expert you know, third party expert eyes on it, because you never know what's in your auto insurance policy until you have an auto accident. And then you learn what it really covers and what it doesn't. So, and I know nobody wakes up every you know,

Robert Cioffi (33:50): on a day on any random day and says, let me learn everything there is to know about this policy. but it's really important to make sure that you do know what's in it. Now, the important thing that did cover us was the breach attorneys, right? Breach is a hard is a terrible word. I tell people don't use it, but for some reason they have breach counsel or breach coaches is the the name for it. So essentially a a team of lawyers was assigned to us. and

Robert Cioffi (34:19): we were able to work with some some technology partners, one of which was Huntress. you mentioned Becky Teal on your previous episode. she and I had been friends and known each other for a while prior to that, but she's also part of the story. she got me connected with a number of people with inside Huntress, 'cause they were very instrumental in doing a lot of forensic analysis work. Kaseya did as well independently, so

Robert Cioffi (34:47): By Sunday morning we had two different independent forensics analysis on our log files, and as well as broad understanding of what had happened to other companies as well. And I'll boil it down to this. While Reval Sodono Kibi, who was the the main operator behind this ty style of attack, is well known for like the double extortion of exfiltrating data.

Robert Cioffi (35:17): up until this point they were well known for pulling data out and then encrypting you. this was a classic smash and grab, meaning they exploited the VSA vulnerability and they pushed ransomware out to as many computers as they could simultaneously with the hope of catching enough things in the broad, wide fishing net that they cast.

Robert Cioffi (35:42): there was an estimate of over 1 million systems worldwide that were affected by that attack. but we knew by Sunday morning, actually we knew by Saturday afternoon by Huntress's analysis, but confirmed by Kaseya by Sunday morning that at 10:49 a.m. the threat actor had gained access to our system, uploaded E agent EXE, ran the script, and by 12:30 p.m. ish.

Robert Cioffi (36:11): so in about ninety minutes ish, one hundred percent of our endpoints were encrypted, right? Now that timeline varied a little differently based on, you know, different company victims, but that's essentially the timeline that had happened for us. So we knew that we also now had an RPO, right? A recovery point objective of of somewhere prior to ten forty nine AM.

Robert Cioffi (36:37): So what we decided to do was just to simply pick 8 a.m. that morning to give ourselves enough of a a cushion there. now we had an amazing

Andrew Moore (36:44): Right.

Robert Cioffi (36:47): disaster recovery vendor we still use today. we had about, like I said, 250 servers that were fully protected by this technology, and we were fairly confident that we would be able to do full restores. but A,

Robert Cioffi (37:06): Well, let me back this up. What we had now, Andrew, was not a ransomware problem. What we had was a math problem. I can take your computer and recover it and restore it either by formatting it if there's no, you know, data needs to be salvaged on it. or if I have a backup of on it, I can, you know, a backup of it, I can restore that, right? and if I had

Robert Cioffi (37:32): You know, all the time in the world, I could probably get you back up and running in a couple of hours, let's say, just you know, spitballing here. I could do the same thing for a server or a customer, right? I've got 25 computers, I've got two servers, let's throw four engineers on this, and before the day is over, they'll be fully restored. But think about the math that I had 80 customers, 200

Andrew Moore (37:56): Right.

Robert Cioffi (37:57): physical locations, four time zones.

Robert Cioffi (37:59): How the insert expletive am I going to get all this work done? And now I want you to think about this. Who do I do first? And that answer is not so much about who you select first, but by selecting who you do first, you're essentially telling everybody else, you're not that important to me.

Andrew Moore (38:20): Right. Pecking order.

Robert Cioffi (38:22): Right? If you got selected first, I must be their best customer.

Robert Cioffi (38:27): And if you're told, mm, we can't get there till Wednesday or Thursday, I must be not important to you. So

Andrew Moore (38:33): Right.

Robert Cioffi (38:34): trust me, the psychological impact of those decisions was not lost on us.

Andrew Moore (38:40): So

Andrew Moore (38:40): I'm a I'm an ops guy, so my first thought would be, okay, we now have our recovery point. We have a pretty good idea. Obviously, like we probably turned Kisseya off, so we're flying a little blind even though we're we

Robert Cioffi (38:52): Yeah. yeah.

Andrew Moore (38:54): have we're so but let's just say that, okay, so we figured out like we're gonna do some recovery, like we're gonna start with our own systems, we're gonna get functional.

Andrew Moore (39:02): We're gonna make sure that we're not, you know, that we're we're turtle shelled in some way. Like we we start there, then we start trying to figure out how to get in, you know, now we've got our base of operations, we're gonna reach out. Did you did it occur to you like that you were like, do I do I maybe I know some people from up peer groups, and maybe like who do I reach out to? Like who isn't impacted? And like, and did it worry you that maybe by reaching out that you had the air of of

Andrew Moore (39:29): being pwned on you and maybe you didn't want like maybe you were embarrassed to reach out to people? Like how did that start to work in your mind? Like they like what do I

Robert Cioffi (39:37): Yeah.

Andrew Moore (39:37): do to start dealing with this capacity issue and how did I move forward with it? Like, what what happened there?

Robert Cioffi (39:42): So here's where here's where

Robert Cioffi (39:43): the word community now slips in quietly, and is the real champion or hero of this story, right? Some people have characterized either me, myself, or our company as a hero or as being heroic, but I'm placing that label on the entire MSP ecosystem and c and community. Community really

Robert Cioffi (40:09): To me, and here's the punchline is community is the missing element to your cybersecurity stack, right? It's the one thing that you haven't installed is community. or maybe you have, but many MSPs have not. and so what happened was it took one of our peer group members, a friend of mine, Kevin, up in Massachusetts, Cape Cod. you know, we're both Northeasterners, so you know, we we punch first and then

Robert Cioffi (40:37): hug and make up late later. Kevin's an awesome individual and he put out a message to our peer group at the time, 'cause my company participates in a group we still do in the same group. he put out a message to them saying that if we don't help out these guys at progressive computing, they're gonna die. And so he basically said hotels

Robert Cioffi (41:06): are waiting for you in Yonkers. book them now. Get on flights. I will pay for your flight. Like Kevin was like really urging people to get here. Now I had spoken to Kevin and initially, you know, because of something maybe we might characterize as a little bit of Italian bravado, said, No no no, Kevin, I like I got this. Like we got this, right? There goes that we got this line again. we got this. Like I don't need your help. Like I was being too prideful.

Robert Cioffi (41:36): and maybe a little stubborn and not maybe realizing that the full magnitude of that math problem that I had urged, but Kevin saw the math problem much more clearly than me. So within like literally about 24 hours of that, he and two other individuals, somebody from Kansas and somebody from Santa Barbara, California, were standing in our office. These are three business owners, and they collectively had brought six engineers with them.

Robert Cioffi (42:06): and like with like no an like a one-way ticket here with like an unlimited hotel reservation. Like they were willing to stay here for as long as possible. all in all, we had 27 companies come to our aid in doing direct delivery of services. And I'm not counting the vendors that helped us, right? Like the Huntresses and the Axians and the Kinectwises and the PAX of the world. And forgive me, I'm probably missing one or two names.

Robert Cioffi (42:35): But we had twenty seven logoed companies coming to help us to help restore our systems over the next seventeen calendar days.

Andrew Moore (42:45): Well, and that that's amazing for the actual MSP community in that like I think one of the most important things that people should remember is that that is potentially a real deterrent to people trying to perpetrate one of these crimes against MSPs in general, because the idea is like well, we'll band together, right? Like we'll like there's almost a deterrent factor against it, not just from a community standpoint to help each other, which is unique in our space, I believe, and in another

Andrew Moore (43:14): industries. But I feel like it's a real opportunity for us to be proud of, you know, what we showed to attackers at the time, which was like, you know, I mean, you mess with us and we're gonna get it fixed and we're gonna get it fixed as quick as we can.

Robert Cioffi (43:25): Well, at least if you if you mess with

Robert Cioffi (43:28): a certain MSP, right, and I and I don't mean this disparagingly to those MSPs who might be sort of living in a little bit of a vacuum or a little bit of a bubble, right? Their their main community connection might be Reddit, right? or some forums. Like you really gotta get out to conferences. You really gotta meet your neighbor, like the guy down the street that you call your competitor isn't really your competitor. He's a friend.

Robert Cioffi (43:55): Right, or she's a friend. make good connections with those people because in a time of need, right, and be a true go-giver with them, right? don't don't stab each other in the back. Make friends in the industry because without those community connections, we would have been dead. My business would have probably dissolved before the end of that calendar year.

Andrew Moore (44:20): So talk to me a little bit about let's let's wrap up the event and then move into we'll call it I'm gonna call it post mortem, but I'm I don't I I that's probably the wrong word. The the path forward and what the

Robert Cioffi (44:32): Yeah.

Andrew Moore (44:33): phoenix that came out of it, not just you know, your business but the the opportunity that you saw to help the community. Let's talk about how this thing wrapped up, how

Andrew Moore (44:43): How you managed to move forward, what clients stuck with you, did they catch the guy? Tell me a little bit about the the end of the story here, and then I want to talk about how you you came out of it.

Robert Cioffi (44:48): Yeah. Well, we we we

Robert Cioffi (44:52): definitely took a big revenue haircut. the way I characterize it is I lost a small MSP, right? and if I know my numbers correctly, if you're probably in that, you know, average sized MSP, maybe a maybe a little bit smaller than you, maybe maybe half your size, that's probably what I lost in revenue. it was a it was a big

Robert Cioffi (45:16): big hit that took place over about 12 months. but I mean we're we're over double our size in that five years since then. So I mean that just speaks to our ability to rebound and all of the customers that did stick with us. I think, you know, it's it's a testament to the relationship that we had with them. but we did manage to get to about a ninety five percent restoration for about ninety-five percent

Robert Cioffi (45:45): of all the endpoints that we managed in those first 17 calendar days. Remember I said

Andrew Moore (45:50): Wow.

Robert Cioffi (45:51): calendar days. We were putting in 18 hours a day running, you know, we were running like lunatics. we had technicians showing up like, Where are you from, buddy? I just met you. Like here's a bag of USB sticks and some instructions and an external hard drive. Like go. Like, you know.

Andrew Moore (46:08): Right.

Robert Cioffi (46:09): we had a a war room set up where Jay and I were

Robert Cioffi (46:13): Like kinda like operating like as commanders or generals in a war

Andrew Moore (46:18): Yeah.

Robert Cioffi (46:18): room, like, you know, constant people filing in and out and like it was the nerve center for our operation. It was exhausting mentally, physically, emotionally. but every day was a bit brighter. That was Kyle Hanslovin, CEO of Huntress, got in my ear on that Friday afternoon. and he told me, he said, I know this is your darkest day, but just

Robert Cioffi (46:42): Hold on to that fight and I promise you that every day is just gonna be a slightly bit brighter. And those words stuck with me and I still you know, appreciate and I tell him all the time how much I loved his encouragement. so that's kind of what got us through by by certainly by the middle to the end of July, we were somewhat holish. I mean, you know, s look, there was still a lot of fires popping up here

Andrew Moore (47:08): Yeah.

Robert Cioffi (47:09): and there.

Robert Cioffi (47:10): you can imagine all the little crap that was still broken, right?

Andrew Moore (47:14): Yeah.

Robert Cioffi (47:14): Printing to the wrong printer tray, missing icons, missing pieces software, settings don't work, like it was just a lot of crap broke, but but we had most organizations just about mostly operational pretty pretty quickly.

Andrew Moore (47:26): Yeah.

Andrew Moore (47:28): Well and that's the part that like the other part of it that really sucks too is you spend years and years and years building up positive capital with your clients, right? And then

Andrew Moore (47:38): it gets burnt on this and so that, you know, I'm sure that any little issue that was outside of the the problem, right? Like any technology problem that normally they would have been considered of, you know, you missed an SLA or the guy didn't do a call back or something, or the project that you were doing for him didn't go quite right. Like normally you could lean on the good graces of your reputation and what you've done to to help him in the past. And I a lot of that got burned to the ground, I'm sure, right? Like

Robert Cioffi (48:07): It got burned, but it

Robert Cioffi (48:08): was also like, you know, when you have something that goes sideways somewhere, CEO gets all pissed off about something. You throw a bunch of great resource or you throw a great resource at it for like an hour or two, and then everything is all happy and joy joy again. But

Andrew Moore (48:22): Really?

Robert Cioffi (48:22): you can't do that when there's a thousand of those requests on your doorstep, right?

Andrew Moore (48:26): Yeah, yeah.

Robert Cioffi (48:28): so it took, you know, that's I think what affected us a bit.

Robert Cioffi (48:34): in terms of some of the losses that we felt that had come later when people decided to go elsewhere. I think it was a combination of maybe that lack of trust or feeling

Andrew Moore (48:40): Yeah. But understandable. Yeah.

Robert Cioffi (48:44): like we did something wrong even though we didn't.

Andrew Moore (48:47): Yeah.

Andrew Moore (48:48): Yeah. That's and I I get that 'cause that's one of those it's like and I'm sure that even though you got all of the good stuff done that, you know, as your contracts expired or whatever, people had an opportunity to consider where their other they've been like, well, you know, it we'll just let's shop around a little just to make sure and then before you know it just yeah, yeah,

Robert Cioffi (49:04): We had some of that too. Yeah. Yep.

Andrew Moore (49:07): I g I get that. So did they ever catch the guy that did it, like or the group that did it?

Robert Cioffi (49:10): Yeah, so

Robert Cioffi (49:12): Yaroslav Vysinski at the time was like a twenty two year old Ukrainian national living in the Ukraine. and he was crossing the border from the Ukraine into Poland, going back to university in October, I think it was October sixth of twenty twenty one. little be known to him.

Robert Cioffi (49:37): the FBI had tabs on him. They were already in their systems tracking this guy. They knew he was one of the primary affiliates in charge of this attack. And you know, if you're not aware that ransomware is is almost like a a franchise business, right? I want to open up a couple Dunkin' Donuts. Well you can go do that, right?

Andrew Moore (49:55): Right.

Robert Cioffi (49:57): Reevil so no kibi the Dunkin' Donuts. I just own these three Dunkin' Donuts like, you know, in this five mile radius.

Robert Cioffi (50:04): so he's this affiliate operator. they called Interpol and Polish authorities, apprehended him at the border, and then within five months had him extradited to the United States. I actually spoke directly to the FBI agent who was there to not only greet him the first time as soon as they had apprehended him, but also had collected him on extradition day to

Robert Cioffi (50:33): bring him back to the United States. And they brought him to Texas, the great state of Texas. I can think of no other better place in the world for a a threat actor, cyber criminal to be e extradited to. and Yaroslav was eventually sentenced because he pled guilty right away. he was sentenced to a total of fourteen years, which included his incarceration in Poland for those first five months. He's now in FCI

Robert Cioffi (51:03): Danbury, so that's Federal Corrections Institute, Danbury, Connecticut, which is about forty five minutes from where I'm sitting right now. and as I am told, once his sentence is up, he will be immediately deported back to the Ukraine. So

Andrew Moore (51:18): Did you

Andrew Moore (51:19): do you ever have a chance to meet him, be a part of the trial, any of that sort of stuff, or is it did you kinda keep your distance?

Robert Cioffi (51:23): yeah,

Robert Cioffi (51:26): interestingly enough that as I, you know, I filed an IC three report as soon as the event took place and you know, I was listening to all those voices in my head of all the conferences that I had been to about things I should do. we had spoken to various FBI agents throughout this. They were more in data collection mode, like tell us what happened, you know. they just wanted to know things. And I was, you know, like I just felt I was doing my, you know, my patriotic duty of like

Robert Cioffi (51:52): Yes, let me tell you as much as I possibly can.

Andrew Moore (51:54): Right.

Robert Cioffi (51:57): and because of that, I got registered in something called the VNS, the victim notification system, and I started getting letters and communications about the trial. and I ended up deciding to go to Dallas, Texas, to federal court to provide a victim impact statement. So

Andrew Moore (52:14): Wow.

Robert Cioffi (52:15): that's something that I did in January of getting my dates crossed now.

Robert Cioffi (52:23): of last year and I for about twenty minutes I spoke to Judge Scholler who oversaw the case as well as the two assistant U.S. attorneys there were FBI agents in the room, defense counsel, as well as Yaroslav Vysinski in a very, I must say, very tastefully orange jumpsuit with these beautiful chrome bracelets both around his ankles and his wrists. so yep.

Robert Cioffi (52:51): I mean and listen, I he he was he was a boy and still is. He's

Andrew Moore (52:57): Yeah. Yeah.

Robert Cioffi (52:58): got, you know, that very cherubic, very young man looking face. Like he

Andrew Moore (53:02): Right.

Robert Cioffi (53:03): looks to me still like a high schooler. and so that's the face of who attacks us. Not the not the guy with the shrouded hooded hoodie where you can't see where their face is obscured, and if they reveal their face like you might see some grotesque monster under it. No. That that

Robert Cioffi (53:20): That's not the face of threat actors. threat

Andrew Moore (53:22): Yeah.

Robert Cioffi (53:23): actors look like you, Andrew. They look like me. They look like this kid. They look like average everyday people, right? And by the way, not to be well, to make sure that I'm not sounding sexist, they're also female, right?

Andrew Moore (53:36): Yeah.

Robert Cioffi (53:36): There's a lot of women doing this stuff too.

Andrew Moore (53:39): Yeah, and and it's you know, all ages, anywhere in the world, it's readily available to them and to your point, the ability to get on the dark web or to

Andrew Moore (53:48): Interact with some of these, it really is a business model that is very sophisticated if people don't understand it, with full market opportunities, kits that can be bought and repurposed to exploit things that are and with AI it's happening faster and faster. And I think MSPs really need to be very thoughtful of how they're protecting their business. And to your point, like you know, making sure that they're integrated into a community that helps them in the event that something were to go.

Andrew Moore (54:18): sideways. And so I know that you have kind of moved past the event and and your your business is now doing very well. Like what are what have you built? Talk to us a little bit about what you built, what came after this that really kind

Robert Cioffi (54:30): Yeah.

Andrew Moore (54:31): of allowed you to tell your story and help others and protect the community a little bit.

Robert Cioffi (54:34): So I got I got very

Robert Cioffi (54:35): introspective. Thank you for that lead in. I I I I got very introspective of August of that year, so about a month after the attack. and I was actually attending, I decided to go, even though it was only a month after. I decided about not going, but then something pulled me to go to a community event. my our evolved community, the very community that helped us. my team was doing a fabulous job, so I felt like, you know, I I could

Robert Cioffi (55:03): I could be on the phone and still still be there. so I got very introspective. Yeah. Yeah.

Andrew Moore (55:08): Well and let's not kid ourselves. Let me pause for a second there.

Andrew Moore (55:13): If people who are not in peer groups don't understand, there's something that only happens in a peer group that you cannot explain,

Robert Cioffi (55:21): Yeah.

Andrew Moore (55:21): which is other people going through the same shit as you in this industry every day, and that gives you this camaraderie, this brotherhood, this sisterhood that people don't get unless they're in one.

Andrew Moore (55:30): So I don't know that you have to make a r a good reason why you didn't go. I'm sure it was really important to your soul and the importance of being able to get in front of a room of people. Yeah, yeah, for sure, I

Robert Cioffi (55:39): I felt like I needed to be there. Yeah. Yeah.

Andrew Moore (55:42): get that. So I and I just want the the listeners to know, like I I think it's important that people find that community and and so I I agree with getting in front of those folks and being a part of that in and you know

Robert Cioffi (55:52): But but because the statistics

Robert Cioffi (55:54): say that there are 50,000 MSPs, I don't know, globally or in this country, I don't know what statistic to believe, but it's quite a number of them. And having attended so many different conferences and peer groups, I know that they're not all there, right? So I was thinking about all the people that I don't know that are not at these events and what would they do if they had suffered something like this? and how would the community even know?

Robert Cioffi (56:20): How would they know who to reach out to for help? And I'm thinking about all their customers and their employees and the associated families and businesses that would get torn down a rubble. How could I help? and so it I had a flash of a thought of registering MSP911.org. So I immediately registered that domain name. It was available. I was a little shocked, but I registered it with this idea that I could be the big giant.

Robert Cioffi (56:50): break glass in case of emergency or the I need help big giant red button or like the fire alarm, right? Like a resource that MSPs could go to should they find themselves hyperventilating like I was, right? Not knowing what to do. Because as smart as I am and as prepared as I am and and as and those words could be used to everybody on my team, we still made a lot of mistakes that day and there was still a lot of

Robert Cioffi (57:18): confusion and you know chaos here. Like there was a lot of a negative emotions fighting all of the things that help prepare you for battle. you can't be prepared for battle unless you're you've actually experienced it. so that's a thing that I wanted to do. Now I'm gonna fast forward to that because it's a multi-year journey there. But we have now rebranded ourselves as CyberRise.org.

Robert Cioffi (57:44): so it's cyberrise.org. It's all one word. There's a double R in the middle. MSP 911 is one of the three pillars of what why we exist and what we do. We are a full nonprofit. that I, as one of the three board members, founding board members, cannot take a penny out of it, can't make a dime on it. Even if I somebody asked me to go speak somewhere, I gotta buy my own plane ticket. I can't, I can't.

Robert Cioffi (58:12): put it through the organization, nor do I want to. but the idea of n MSP nine one one is that if you're suffering an event, you can call that phone number or send an email or use the web form to say, I need help. And somebody like me, and half the people that are helping with this right now are former victims like me who know what you're going through and can provide those coaching and assistance type of

Robert Cioffi (58:44): services to you to help you step through that that process so that you are not going through it alone. we just at PAX 8 Beyond in Salt Lake City in early June, Matt, Lee, Jason Slagle and I, that we are the three founders of CyberRise, had case 50, a gentleman by the name of Michael

Robert Cioffi (59:09): who was running an organization serving only dental practices that at the beginning of this year had gone through I'm sorry, at the end of last year had gone through a very horrific similar event. And he has said multiple times, I would not have survived if it weren't for you guys. Like you guys were like angels. You came out of nowhere, right? I think a friend of a friend connected him to us and and it's like it's a miracle to him in his own words

Andrew Moore (59:36): Right.

Robert Cioffi (59:37): that his business is still alive.

Robert Cioffi (59:39): Because we rendered assistance to him.

Andrew Moore (59:41): No, I that's awesome because I was at that PAXE conference and I didn't know of you or that you were there. And so the fact that this exists is fantastic and we need to get the word out to everybody 'cause, you know, I it I'm sure that there's things that in the event of a breach,

Andrew Moore (59:59): you know, the it's awesome to be able to reach out, but I'm I'm guessing there's probably just some information out there about like how to be better prepared and what to think about and like not just like, you know Yeah, yeah, 'cause

Robert Cioffi (1:00:09): We're we're doing that as well, yes. Yeah.

Andrew Moore (1:00:14): Having an incident command structure and knowing who's in charge during one of these issues and some of that stuff is just some to your point earlier, it's just like some basic stuff that like you don't even think about, like you just kind of go into React mode, but if you've already got some plans and some ideas and I know that you know Becky talks about doing tabletop exercises when it comes to client breaches, like but if you're to your point earlier, if you're not eating your own dog food, if you don't know how to manage something within your own environment, I think it it it really behooves you to get prepared for that even prior

Andrew Moore (1:00:44): to knowing an organization like you guys are out there after the fact.

Robert Cioffi (1:00:47): So

Robert Cioffi (1:00:47): wanna highlight something really quick. You were kind of alluding earlier that like the threat actors don't know who they're messing with when they're messing with MSPs. This was the core tenant of what I saw as the underlying thought behind MSP nine one one is that I don't like emotionally I wanted to get on an airplane with, you know, fully armed and go f seek my attackers, right? That's what the emotional Robert was thinking.

Robert Cioffi (1:01:13): I actually had a few friends that were We got I we got you, brother. I got the ammunition. Let's go. Like

Andrew Moore (1:01:17): Let's go.

Robert Cioffi (1:01:18): I'm like, yeah, it's it's a wild fantasy. That's not gonna happen.

Andrew Moore (1:01:21): Right.

Robert Cioffi (1:01:24): but I did think, well, how can I fight these guys? They're smarter than I am. They have they're they're technically smarter than me. They have more resources than I have, they are much more motivated than I am. What can I do to

Robert Cioffi (1:01:39): fight them and this is how I'm fighting them is by being the tide that lifts all boats, by educating the community, driving the awareness, connecting community together, and providing this resource that helps someone who is in that shit moment and

Andrew Moore (1:01:54): Right.

Robert Cioffi (1:01:55): we could potentially turn it around for them and we did.

Andrew Moore (1:01:58): I love that. And so if they if anybody runs into an issue, they can simply go to the website and reach out to you or they can, you know, you know, we'll have information like your LinkedIn profile will be listed here too, so they can reach out to you directly or somebody else. Yeah.

Robert Cioffi (1:02:12): There's a simple web form, you fill it out,

Robert Cioffi (1:02:14): it opens up generates a ticket in our s our incident response platform. We all get notified and you know, we take the call from there. Yeah.

Andrew Moore (1:02:23): I love that. And then you're at

Andrew Moore (1:02:24): a lot of the conferences. I'm guessing you're gonna be at IT Nation in Orlando later this year and you're probably

Robert Cioffi (1:02:28): Connect. Yep. Yep.

Andrew Moore (1:02:30): at sounds like you went to PAX 8 and so you're gonna be around. You're in the channel.

Robert Cioffi (1:02:34): I

Robert Cioffi (1:02:34): try to get out to as much that I think will benefit our business as much as possible. So and hey, look, you know what I talked about Evolve. I mean, they're obviously owned by ConnectWise. I just want to be completely, you know, agnostic here. There are a lot of other great peer groups out there. It doesn't have to be Evolve. I love Evolve. you might love something else. Just get the to a peer group, right? I think Shakespeare said that. Get the no, I think he said something else, but

Robert Cioffi (1:03:02): or he said get thee to a nunnery, I think is what he said. But

Andrew Moore (1:03:05): Yeah.

Robert Cioffi (1:03:06): I'm saying get thee to a peer group. Like I don't care which one you pick, like go to one. Get yourself involved in one. Plug in, please, please.

Andrew Moore (1:03:11): Right. Yeah. Yeah.

Andrew Moore (1:03:15): Yeah. And and I the the podcast we just released recently with Becky, you know, and I and I I stressed this and this was a a big topic that we talked about with her is

Andrew Moore (1:03:24): If you've got partners, like if you're working, like you call them vendors, like I like to call them partners, if you don't treat your vendors like you would treat, like you want your clients to treat you, if you're not treating them that way, if you're not going to these shows, if you're not reaching out to them and creating personal relationships with them, not just when you need something or you're dissatisfied, but you're going to the show, you're sitting in the booth, you're having a 15-minute conversation, you're lining up a business review for after the show, you're having drinks with them later, right? If you're not doing those things,

Andrew Moore (1:03:55): Then you don't really know what resources they have available to you. You don't understand how

Robert Cioffi (1:03:58): Correct.

Andrew Moore (1:03:59): important their relationship can be to your business. And it's not just being at the peer group, but it's being integrated into the channel because there are a lot of things that are happening out there that if you're not aware of it, are really going to be beneficial to your business and don't cost you a penny. They don't cost you a dime. And you should be out there. And even peer groups are fairly inexpensive when you look at it. So I I encourage it.

Robert Cioffi (1:04:20): Yep. for the value

Robert Cioffi (1:04:23): that you get out of them, I think there's tremendous value. you know, and it's be a go giver, right? If you haven't read the book

Andrew Moore (1:04:29): Yeah.

Robert Cioffi (1:04:29): The Go Giver by Bob Berg, I would highly implore you to read that book. but it's more about like I try to go to these things. I mean, I know they're trying to sell me, but I'm like, how do I help you? Right? I'm dealing with a vendor just a solution partner just coming out of their skunk works, right? supposed to

Robert Cioffi (1:04:49): I think I'm still under NDA, so I can't say who it is.

Andrew Moore (1:04:51): Yeah.

Robert Cioffi (1:04:52): But like I'm one of 10 design partners and I'm looking at this like they're saying, We want to help you. And I'm like, no, I want to help you because the more I

Andrew Moore (1:04:58): Right.

Robert Cioffi (1:04:59): help you get stronger, then I know that it comes back to me later. So that's the mentality that you have to have is I'm not there for me, me, me, me, me. I'm there to like, how do I help? Right.

Andrew Moore (1:05:12): Yeah, yeah, I love that. It's a it's the the right attitude to have. And so, I this has been a fascinating conversation. I know it's been a busy day and I don't want to keep you any longer. We got a couple of questions I wanna ask you before we let you go, but I just I appreciate you sharing with us and I think it's been a fantastic journey to to to hear that you came out of

Andrew Moore (1:05:33): Well, I'm not gonna say unscathed, but w having learned something incredibly important and not only done that brought the resources that you have to bear on the channel to make this community a better place. And so I think we're all better off for you having been strong.

Robert Cioffi (1:05:46): Andrew, I I might I might joke around a little

Robert Cioffi (1:05:48): bit, I might, you know, seem kind of jovial about it, but please understand and and anybody listening or watching should understand that the PTSD that I have is real. I have not sought professional help, but I have spoken to professionals about what I went through and they have told me one hundred percent you have PTSD.

Andrew Moore (1:06:09): Yeah.

Robert Cioffi (1:06:09): I have also spoken to other victims just like us, and it's real.

Robert Cioffi (1:06:14): And it lives with you, it stays with you. Even though it's five years later, I still I still, you know, I get that anxious feeling, but this is a coping mechanism for me. It's cathartic to be able to talk about it. And if I get one person that reaches out to me and said, you know what, I'm taking some additional steps to better my company, that's well worth what happened to me.

Andrew Moore (1:06:35): Well and I I'll finish by saying I'm gonna butcher this, but the antidote anecdote to the antidote to anxiety is action. And so if you are anxious, if you were uncomfortable, get out there, do something about it, make a change. so I think that's great. you mentioned the go-giver, let's let's get into the questions real quick and I'll let you I'll cut you loose. What is your what is your favorite business book? What's the one book or books that you're like, you gotta read this, this is amazing, this is gonna help the business.

Robert Cioffi (1:07:00): Yeah. man. I I I have so

Robert Cioffi (1:07:05): many and it was so difficult for me to think about what the winner is here. So I'm gonna pick on one that's definitely in my top ten. I'm not saying this is my top, but how to win friends and influence people, especially if you're in mentoring somebody else or you're leading a a company, I would say give that book to a twenty to thirty something year old, like you know, someone who's early in their career.

Robert Cioffi (1:07:30): even those that haven't read it is just tremendous value. I didn't read that book, I think, until I was about forty. so I think that's an amazing book.

Andrew Moore (1:07:40): That was Dale Carnegie, right? Yeah, yeah,

Robert Cioffi (1:07:42): Dale Carnegie, correct.

Andrew Moore (1:07:43): great book. Great book. Awesome. and then what's your favorite curse word? What do you what do you like what do like to use? Like what do you feel comfortable saying on the podcast?

Robert Cioffi (1:07:53): well I I used a couple of already, but

Andrew Moore (1:07:55): Yeah.

Robert Cioffi (1:07:56): I'm gonna go to my my heritage, my my my my my DNA here being Italian. The word is cazzo and it means so many different it it's the F word, right, in Italian

Andrew Moore (1:08:05): Okay? Okay.

Robert Cioffi (1:08:07): and and it's used in all the different ways that you can think. it is it's it's just like our F word in the various ways you can use it. But

Robert Cioffi (1:08:16): it's Italian, so it sounds more passionate and it's more powerful. So but if my

Andrew Moore (1:08:19): Yeah. Well and and

Robert Cioffi (1:08:21): mother heard me say it she'd slap me, so

Andrew Moore (1:08:23): Well in in the most Italian way possible too, for those who were w who were just listening, you used your hand very like emphatically when you used the word. So I love that you were very So good.

Robert Cioffi (1:08:33): like doesn't everybody speak with their hands? and it's

Robert Cioffi (1:08:38): C A Z Z O, right? It's a that hard Z. So anyway. Yeah.

Andrew Moore (1:08:41): Yeah, that's a good one. That's a first. Like

Andrew Moore (1:08:44): we've we've we've had we've had some British cursing on here before, but not not and I is it they do speak English, but you know if you're you've heard British slang, they've got some very you there's a unique podcast out there. Yeah.

Robert Cioffi (1:08:57): Specific words, yeah.

Andrew Moore (1:08:59): favorite band. So if you're into music, like who who you're on a desert island, band or artist that you're like, I

Robert Cioffi (1:09:05): Yeah.

Andrew Moore (1:09:06): gotta this is my go-to.

Robert Cioffi (1:09:08): this is also a really, really tough choice, but I'm gonna go with Pink Floyd. my

Andrew Moore (1:09:13): Okay?

Robert Cioffi (1:09:14): all-time favorite album is not only my favorite song of all time, but my favorite album is Wish You Were Here. it is what motivated me to learn how to play acoustic guitar. Now, for all of you fans out there, trust me, I play acoustic guitar badly, so you don't want to hear me play. but it is the one song that I can play without having to look at a music sheet.

Robert Cioffi (1:09:38): and I'm proud to say that, but I'm still sort of at the beginner level of playing. I just that album and w between Wish You Were Here, and some of their other albums, I just i

Andrew Moore (1:09:52): Yeah.

Robert Cioffi (1:09:53): it Dark Side of the Moon, like I can get on an airplane and get completely lost in that music.

Andrew Moore (1:09:58): Yeah.

Robert Cioffi (1:09:58): It is it's I I just find it takes me to a different place.

Andrew Moore (1:10:03): And you're you of the almost twenty podcasts that we've done this year, you're like the fourth person that's either picked David Gilmore or Pink Floyd as their favorite

Robert Cioffi (1:10:12): Yeah.

Andrew Moore (1:10:12): artist. So and I'm either I've got a demographic, either I've got a bunch of you know old white guys right that are that are now like, you

Robert Cioffi (1:10:17): Got a lot of Gen Xers. Yeah. Yeah. Yeah.

Andrew Moore (1:10:20): know, having that conversation, but it seems to be pretty prevalent.

Andrew Moore (1:10:23): I remember the first time I heard the wall, and I just was like, my mind melted, and I was just like, What is this? so

Robert Cioffi (1:10:29): Yeah.

Andrew Moore (1:10:30): love it. That's great. okay. Worst sales call, business meeting, personnel meeting like that you've ever had. Maybe something

Robert Cioffi (1:10:38): Yeah.

Andrew Moore (1:10:38): that's a little funny, but something that you're just like, dude, this was crazy. You gotta hear this.

Robert Cioffi (1:10:41): Well, this this is

Robert Cioffi (1:10:42): a story. It goes way back in time when we were still doing hourly break fix work. our MSP services were starting to come up. We were still doing a lot of break fix and I had this woman call me and she was complaining about all of the IT people that she had just gone through and that probably should have been my first hint, by the way. they're not returning my calls. This guy disappeared for two weeks and

Robert Cioffi (1:11:07): And on and on and on for a literally about 20 minutes. And I'm just sitting there like playing therapist, listening to her complain all of the bad work, the terrible things that they've done. and she just wants to find somebody who's really a true professional that's just gonna get things fixed. And you know, so I finally was able to cut in edgewise and like, okay, so let me tell you a little bit about us and how we work. And one of the things I started to step through was our

Robert Cioffi (1:11:37): Our rate and our, you know, some of our billing practices just so that, you know, just to drive some general awareness. and she was like, What? Like I think at the time it was like we were charging $150 an hour. I might be off on that, but that was probably about the rate. and she said, What? How do you dare charge me $150 an hour? And I said, Well, how much were your three previous guys charging you? And they were like, Well, they were all between forty and fifty dollars an hour. And I said, Well, there's your problem right there.

Andrew Moore (1:12:07): Right.

Robert Cioffi (1:12:07): And I can tell you that she just never called me back, so anyway.

Andrew Moore (1:12:11): That's

Andrew Moore (1:12:12): that's a that's a I love that conversation. Like there there it's funny, I one of our previous guests from Ashton Tech, Travis, would tell me this story. And he would one of the questions they ask is in the in the pre-sales process are like, how many MSPs have you had in the last three years? And if the answer is over is three or more,

Robert Cioffi (1:12:35): Yeah.

Andrew Moore (1:12:35): then you know that there might it it might not be you at

Robert Cioffi (1:12:39): It might not be the MSP,

Robert Cioffi (1:12:40): it might be the Well t I mean what I also

Andrew Moore (1:12:40): Yeah, yeah, exactly. So

Robert Cioffi (1:12:43): s told her was like you're focused on the rate, you gotta focus on the invoice amount at the bottom because what we might do in two or three hours at a buck fifty an hour, they they're taking eight hours to do. And by the way, they're doing a shitty job and we're gonna

Andrew Moore (1:12:54): Right. Mm-hmm.

Robert Cioffi (1:12:57): do it professionally. Like I I I'm not sure what you want. Like

Andrew Moore (1:13:01): Yeah.

Andrew Moore (1:13:01): Yeah. I had I had a very similar conversation with some client and they like, you know, how much are you gonna charge us? I think we were like, Well you're gonna be about three grand a month or whatever.

Andrew Moore (1:13:08): And he kinda freaked out and like, What you paying now? And he's like, Well I'm paying twelve hundred bucks. And I'm like, Do you like the MSP that you're working with? He's I like but they don't do this other stuff. And I I was like, Well look, man, show me your bill. And I looked at it and I was like, Do they have like an all you can eat service offering? He's Yeah, I think they do. And I'm like, Probably gonna be about three grand a month, and he's like, Yeah, and I was like, Man, just go back to them and tell you want the upgraded plan. You don't need to change. I'd love to take your business, but man, like, you know, you just pay more. Like sometimes paying more is the answer, right? It's just

Andrew Moore (1:13:38): Yeah.

Robert Cioffi (1:13:39): It it I I have learned that as well. You want you want you

Robert Cioffi (1:13:42): want performance, you gotta pay for it, right? you want quality, it comes at a price.

Andrew Moore (1:13:45): Yeah, yeah. We

Andrew Moore (1:13:48): we tell people we're like we may not be the cheapest people, but you know, we're we're gonna do a better job. so last question is MSP owner, operations person, person in the channel, partner, vendor, who is somebody that has a story that needs to be told that we should be talking to? Is there somebody you can make an intro to? Who you who you know?

Robert Cioffi (1:14:04): Yeah. so

Robert Cioffi (1:14:06): again, another one of those things where I've got like such a big catalogue of people I could point you towards, but I thought about Dustin Bolander. he is out of Austin, Texas. I don't even

Robert Cioffi (1:14:18): want to tell you his company name because he's running multiple companies.

Andrew Moore (1:14:23): Okay.

Robert Cioffi (1:14:24): he's you know, he's a bit younger than me. I don't think he has any gray hair on his head yet. so I hate him for that.

Andrew Moore (1:14:30): Ha ha

Robert Cioffi (1:14:31): so

Robert Cioffi (1:14:31): Dustin, I hope you're listening because I hate you for that. No, I'm kidding. Dustin is an awesome guy, awesome friend. We talk regularly. I think he's got some uniquely insight fullness about or insights into the cybersecurity space. So he's

Andrew Moore (1:14:47): Okay.

Robert Cioffi (1:14:47): not only an MSP operator focusing on legal, but he's also got a tremendous wealth of knowledge about cybersecurity and you know all of the

Robert Cioffi (1:15:00): The things behind the scenes that you wish you knew, you would

Andrew Moore (1:15:03): Yeah.

Robert Cioffi (1:15:03): talk to Dustin and he will tell you.

Andrew Moore (1:15:06): I I would nerd out with him because I in my former life I before MSP I worked on a grant from Homeland Security.

Andrew Moore (1:15:13): back early like when it first got formed like early 2002, 2003, I think, maybe was it right after it came out and bate when they created Homeland Security. And we started as this bioterrorism defense group inside of the the city of Houston under a federal grant. And man I learned so much about incident response practices and like before cybersecurity was like kind of a thing. And they were like how

Robert Cioffi (1:15:35): Thing, yeah.

Andrew Moore (1:15:36): do you get together, how do you talk, how do you communicate, how do you build processes? And so it's come so far that I'm beyond knowing

Andrew Moore (1:15:43): all the ins and outs of it anymore, but I really still nerd out on it because it's just such a special part of like how I learned IT as an engineer and so I would I would love to talk to somebody like that. I'd love to get an intro. That'd be great.

Robert Cioffi (1:15:54): Yeah, yeah. So he owns an MSP, he's got an insurance

Robert Cioffi (1:15:57): company, he's got his hands in a few other things. That's the other thing I really like about him. He's a true visionary.

Andrew Moore (1:15:59): That's great. Yeah.

Robert Cioffi (1:16:02): and but he's a visionary that executes and I think that's a rare b breed.

Andrew Moore (1:16:07): Well he's I'm in Austin too, so if he's here I'd love to meet him. So I'd love to meet people in my own backyard. But

Robert Cioffi (1:16:10): Yeah.

Andrew Moore (1:16:13): hey Robert, I can't thank you enough for your time today. You've been more than generous. I appreciate everything that you are doing for the channel and sharing your story here with us and I look forward to seeing you in person at IT Nation at the absolute minimum. All right.

Robert Cioffi (1:16:26): Can't I can't wait to meet you in person, Andrew. Yeah. It'll be it'll be great.

Andrew Moore (1:16:30): All right, thanks. Have a great weekend, sir.

Robert Cioffi (1:16:32): Thank you.