From LexBlog:
We predicted last year that hackers would become more malicious in the future, not only stealing and selling data for nefarious purposes, but actually destroying data and even systems. That reality hit email provider VFEmail last week, and on February 12, founder Rick Romero tweeted “Yes, @VFEmail is effectively gone. It will likely not return. I never thought anyone would care about my labor of love so much that they would want to completely and thoroughly destroy it.” The tweet went out after he watched the intruder reformat the hard drives of his email service, which has been in existence since 2001. The intrusion wiped out two decades of data. This is a tragic story.
Link to the rest at LexBlog
You may not need to keep the meandering emails about the old days from Uncle Fudd that make their way into your inbox all too frequently, but, if you’re a professional writer, you probably have some business emails sitting there as well.
PG has used Thunderbird as his email software almost forever. After reading the OP, he backed up his important emails to a couple of different locations using Thunderbird’s export function.
He’s also going to try a couple of third-party email backup solutions as well.
One of the practices that make PG’s backup job easier is that he has Thunderbird set to route important emails to some specific folders, so he knows where to go to find good files to backup. He also uses different email addresses for different purposes. (Thunderbird can handle multiple email accounts without any problems.)
For example, if you’re wandering around online and sign up for some update service you’ve never seen before, until you learn whether the updates are going to be really useful for you, you might want to use an email address that is just for those online sites that haven’t yet demonstrated they won’t fill your inbox with junk.
The last time PG checked, Gmail and Outlook.com were still providing free email services. You won’t be able to get bob@gmail.com because that was gone a long time ago, but when PG just checked, bobthehackwriter@gmail.com was available. He has 3-4 Gmail accounts he uses for different purposes. If you don’t want to check 10 separate email boxes all day, you can set most email services to automatically forward any incoming emails to another address, so all your separate email addresses funnel into a single place.
If you have multiple emails, it’s not a good idea to use the same password on each one. (Ditto for almost everything else you do online). To keep the various id/passwords straight, Windows will offer to save them, but PG prefers LastPass, which offers a free version that works just fine. He has also heard great things about 1Password. If you use more than one computer or a computer and a smartphone, a password manager like LastPass syncs all the passwords you store in it to your LastPass account wherever you install LastPass. When you trash the old computer and get a new one, LastPass will bring all your passwords and other secret stuff to the new computer as soon as you install it.
For client emails, in addition to the mass email exports to backup locations, PG often simply prints important client emails to PDF, then stores the email in the appropriate client file folder. It’s much easier to do a quick check of emails that way than to dig through a giant email archive. His client files are backed up seven ways from Sunday, locally and remotely.
PG invites one and all to post comments with their backup solutions for important emails in the comments.
Jake on 02-23 makes an excellent observation: at some point everything electronic will fail. That’s why I have an external hard drive redundant to my main computer plus a cloud back-up account with SpiderOak.
To the email point — I did discover at some point I was keeping a lot of worthless stuff (I really must read this article…from 2011!?) — but the stuff I want to keep I export to Eagle Filer (which I highly recommend if you’re on a Mac) https://c-command.com/eaglefiler/ …. Easy to use; easy to search.
Been an IT guy for a LONG time. Used to have my own server, but the constant fight with spam a decade ago meant I have long since abandoned my private server.
I use gmail and outlook.com for everything now. No reason not to.
But if you DO have your own server. BACK IT UP. OMG, that dude has been doing this for 2 decades and never backed his data up to tape? Or an offline ______ (something)? He had no business having an email service, and I hope no one besides himself suffered due to his data loss.
In any case, another VERY easy backup for important files is to use Onedrive and Google Docs. Either one is free and is enough for typical usage for any but the most unbelievably prolific writers. Just save your documents there, AS YOU WRITE THEM. You then don’t need to think about scripts or copying them over once a week or nightly, etc.
To reiterate: In Windows 10, look for Onedrive in Explorer, ensure you have a Microsoft account of some sort, and when you create a new document, ensure that it’s saved in the Onedrive folder.
If you have an Apple machine, I believe Apple has been offering a similar service for an even longer time than Microsoft and Google have.
I’vd been using dedicated email addresses since the 1980s. I find it very useful as a kind of first cut organizational system.
For passwords, I use 1Password which is a godsend. In addition to passwords on all my devices, it keeps track of credit card numbers, software licenses, & other number based info, & reports whether any of my passwords have suffered from internet security breaches. And it does a pretty good job of entering login info onto sites based on facial recognition or fingerprint.
Level 1: External hard drive for backup whenever a vital file is closed.
Level 2: Daily backups to a DVD-ROM. When full, that goes in the fire safe.
Level 3: (Which I don’t do nowadays, thanks to a bit of financial tightness) – all but the latest DVD-ROM to a bank deposit box.
I’ve never used more than level 1, to be honest.
Oh, and “Level 0” – any truly vital files are “backed up” – via a device called a “printer” with the number of copies set to 3.
3a: Online. Though encrypted if you don’t want others reading it. And since you don’t ‘own’ the system it’s stored on, it may disappear one day.
I back up my entire system to an external USB drive every month. I rotate backups among three drives; two are in a different building, one goes over to a friend’s house. I do incrementals to thumbdrives whenever I feel the urge.
Remember: backups are nice, but only restores count.
So ’email provider VFEmail’ is admitting that they placed no value in those emails – as they had no backup/recovery solution in place.
So if it wasn’t ‘hackers’ a stray lighting bolt, power surge, fire or flood would have also have destroyed all that data.
One other possibility of course is ‘operator/user error’ – which now a days can then be blamed on those nasty ol’ hackers.
Funny that it’s been up since 2001 because I was working at Dell that year.
On one of the calls I got the owner of a brand new Dell sever was calling in. Seems he’d had a couple consultants come in and set his server up for him and move his database systems over. He was calling (on a Sunday) because he’d been ‘working’ on the server and it had suddenly stopped working, and on reboot it was claiming to not be able to find its operating system. A few questions about ‘what’ he’d been ‘working’ on let me know he’d decided to change the partition sizes of the drives to better reflect what he thought he’d be using. (For those that ‘use’ their computers and don’t make major changes, any/all changes to partitions under Windows NT4/2000 required reformatting the new partition – wiping all data from it.)
When asked about backups, he said that that was what he was going to do ‘next’. Too bad there was nothing left to back up any more.
Do as I say, not as I do; for I too have lost data – some of which I ‘thought’ I’d backed up … 😉
> So ’email provider VFEmail’ is admitting that they placed no value in those emails – as they had no backup/recovery solution in place.
they didn’t have offline backups, but that’s not uncommon, especially if you are dealing with a ‘cloud’ service.
In this case, the hackers wiped all servers, including the backup servers
Okay, I’m going to need to go get the tartar sauce …
Unless they don’t know what they’re doing (possible) or their ‘data backup servers’ were also tasked with other things related to or requiring internet access (also possible), the backup severs don’t communicate with anything but the servers – and then only to go get the data to back up – so no access to the OS by the hackers.
Of course I’ve seen far too many people that should have known better think that having mirrored drives or a RAID array was a ‘backup’. For those wondering, a ‘backup’ is a second (or more) copy NOT easily accessed by the system running/using the data. Automatically saving a file to two different drives is nice, but it doesn’t help if both those files are just as easily changed/overwritten/deleted. If you copy your docs to a SD card at the end of each day that’s great, but pulling the card when not coping files is even better. That was an advantage of the old tape backups (I say old but still used in many places), you had to start the backup software and load a tape (yes, could and was automated), and you had an ‘offline’ backup in case Bob (everyone has a Bob – right?) did some Spring cleaning and killed off some of those old folders – that happened to have some files you still use/need.
From the OP:
“According to Romero, the damage the intruder inflicted included VFEmail’s entire infrastructure, including mail hosts, machine hosts and an SQL server cluster, which led him to believe that the intruder had multiple passwords when hacking into the system.”
No mention of backups, no mention of the backups turning out to be bad/corrupted.
And once again for those not ‘into’ all this, a ‘cluster’ is two or more servers serving from a common data array, sometimes in a load sharing/balancing mode, sometimes as a backup where if one server fails the other/next one takes over. Which is great for up-time (no/less customer side outages), but it does nothing in the way of actually protecting/backing up the data in question.
I’ve seen several other stories on this incident (Computer Security is my day job)
They did have backup servers, but all their systems were virtual machines hosted on physical servers owned by someone else and the hackers were very thorough and wiped every system they managed, including the backup servers.
(And walking Admins through building/changing/adapting/repairing/rebuilding their servers and data arrays used to be my 21:00-08:00 job. 😉 )
What I was getting at was there was no ‘offline’ storage/backup – if there had been the hackers wouldn’t have had access to it and there’d be something to recover. So ‘if’ they had any actual backup it was as online as their regular database – and as easy to kill it seems.
So virtual machines with virtual backups? Ouch.
All their eggs in one basket and someone bumped their elbow (if that’s really what happened and someone didn’t lose their own data by trying to rebuild a failed RAID array by replacing the wrong drive – got to watch that happen more than once.)
I agree that a lack of offline backups is a mistake, but it’s different from “they didn’t care about their customers”
Unfortunantly, there are a lot of companies that are not making offline backups.
Especially if you are running ‘in the cloud’, it’s expensive to get data out of the cloud to something that’s offline, so a lot of companies don’t do it.
Which circles us back to that the data wasn’t important enough for them to take those extra steps – there wasn’t enough ‘value’ in the data for them to bother with it.
And I have noticed a few signs that some companies are starting to figure out that ‘the cloud’ isn’t all it’s cracked up to be.
With a local server/admin you have someone with a vested interest in making sure you and your data stay secure and is accessible – it’s there one and only job – and screwing up can get them fired.
With ‘the cloud’ the vested interest in you and your data isn’t there, nor is there someone the boss can really yell at to speed things up (never mind your entire staff having nothing to do because your cloud or internet service is offline for any reason.)
Heh, before ‘the cloud’ we already had companies ‘remote’ servicing more than they actually knew how to. I had a call for a backup problem. No big deal, a few tests and I’d send out parts/tech as needed. Except the calling ‘admin’ wasn’t allowed to touch the server – other than to swap the tapes when told to. And I wasn’t allowed to bother their ‘remote’ admin about it. Icing on the cake was that the caller had never even heard of cleaning tapes before I asked how often they ran one – I was told that if cleaning was required then Dell should have been coming out an cleaning it all along … (I checked a week later and they hadn’t called back, so I guess someone figured it out. 😉 )
They were being incredibly irresponsible. I just hope VFE weren’t charging for their service. Offline backup is an absolute must for anyone who has money changing hands. Even if you run your servers out of a data center, you can have automated tape machines that will rotate tapes for you. You can have local staff remove tapes. (Which should be done on, at most, a monthly basis…) – this is Data Security 101 for anyone who calls themselves an IT professional with a modicum of seriousness.
FWIW, it likely wasn’t a hacker group, etc. It was likely a former employee. (According to IBM Threat Intelligence, 60% of hacks are still former employee related…this one REEKS of it…)
The fact that the attacks “originated from Bulgaria” simply means they used a VPN to mask their IP address.
And if it WAS a former employee, all admin passwords should have been checked and changed as soon as they left the company.
Again, Data Security 101.
AD (After Dell) I was with a small firm that had taken over management of several companies IT departments. There was one company that kept having problems with their network. Funny thing was there were still dozens of ‘users’ on the domain that no longer worked there – many with admin rights. After a little chat with the company owner we changed the passwords of every single user on the system, and spent a Monday morning confirming who actually needed what right to what systems/data and having them reset their passwords. Never had another problem (the guess was an ex-admin was trying to prove they couldn’t get along without them.)
Outlook, since forever (my career was in IT). I back up the Outlook mail folders locally on harddrives, and I have an elaborate email folder system within Outlook for my 2-3 decades worth of email to be kept. (All my old business arrangements, and some personal stuff.)