My worst nightmare as an IT Admin just came true tonight

Demon_Skeith

Demon_Skeith

Administrator
Staff member
Administrator
Credits
47,448
Full year of Nintendo Online
Steal Penalty
You're Rich Money Bags Award
Profile Music
Strange shit had been happening to our stuff in the morning today, all of a sudden our Vms all restarted, some came back up, some did not while all my veeam backups vanished. We thought it was one thing and roughly near the end of the day, VM support confirmed we were being ransomware attacked. Saw my harden CIO pale and shake for the first time tonight as we started to scramble to get stuff back up and offline.

Needless to say, I'm working all this weekend :squint:
 
Dang! Hopefully you will be able to get stuff back up and online soon.
 
If you know what you need to do then keep doing it and who tf thought they should start a ransomware attack?
 
cm2 said:
Dang! Hopefully you will be able to get stuff back up and online soon.
Click to expand...

We got some stuff backup, but until the forensics team can look our network over, we can't go forward as much as we need to.

Grungie said:
What OS are your VM’s hosted on?
Click to expand...

Mixture of Windows and Linux OS

Artisan Vistra said:
If you know what you need to do then keep doing it and who tf thought they should start a ransomware attack?
Click to expand...

Sadly we lack the equipment to really get going, spent half the day getting pieces of crap trying to work. And when it did work it fought us in trying to install new server OS.


Currently we're waiting on the final results from the SentinelOne team who is reviewing our network and scanning our machines for infection. Early results show machines infected and we have 70 laptops coming in tomorrow brand new that need to be prepared.

Going to be so nuts tomorrow.
 
Hopefully, there is not much data lost, and your IT department can fully recover from the Ransomware attack.
 
Is this infecting only the host OS's, or just the VMs? If it's the host, is it just on the Windows host? If it's the Windows host, I'd swap that over to a Linux distro.

Our hosts run on ESXI.
 
Our stuff is barely coming back up thanks to old equipment meant to be thrown out, some old development programmers that are reaching their breaking point and a handy team from RSM that I wouldn't begin to want their job.
 
Demon_Skeith said:
Our stuff is barely coming back up thanks to old equipment meant to be thrown out, some old development programmers that are reaching their breaking point and a handy team from RSM that I wouldn't begin to want their job.
Click to expand...
This is why upgrades are very important. This issue might have been easier to fix.
 
Heatman said:
This is why upgrades are very important. This issue might have been easier to fix.
Click to expand...

Can't fix infection when you can't have down time.

But holy fuck am I done with this week. We at least got vpn up so I don't have to go into the office again this weekend, but got to keep a close eye on restores till they get done. Really fucking came close to quitting today.
 
Demon_Skeith said:
Can't fix infection when you can't have down time.

But holy fuck am I done with this week. We at least got vpn up so I don't have to go into the office again this weekend, but got to keep a close eye on restores till they get done. Really fucking came close to quitting today.
Click to expand...
It's one thing at a time.....and hey just try as much as possible to keep that quitting thoughts off your mind.
 
Demon_Skeith said:
Our stuff is barely coming back up thanks to old equipment meant to be thrown out, some old development programmers that are reaching their breaking point and a handy team from RSM that I wouldn't begin to want their job.
Click to expand...

Oh damn! That must have been one hell of a week for every member of the team. Energy draining and mentally exhausting, I must say.
 
You must log in or register to reply here.
Back
Top