Blog – Defiant Technologies, LLC

The problem of recovering data from SSD drives

Post author By Dmitry Brant
Post date May 24, 2022

One frequent question I receive from users of DiskDigger is: Why does it seem to be unable to recover data from my internal SSD drive? And since SSDs have become nearly ubiquitous in laptops and desktops, this question is becoming more and more common.

The short answer: It is generally not possible to recover deleted data from internal SSD drives, because they are very likely using the TRIM function.

How do I know if TRIM is enabled?

It probably is. If you have an SSD drive that is internal to your computer (NVMe drive, SATA drive, etc), and you’re using a modern operating system (Windows 7 and newer, macOS, etc), then it’s likely that TRIM will be enabled by default, because it’s highly beneficial to the performance of your SSD drive.

Why?

SSD (flash memory) drives work fundamentally differently from older magnetic (spinning disk) hard drives.

With both types of drives, when data is deleted, the physical blocks that were occupied by the data are marked as “available”, and become ready to be overwritten by new data.

With a magnetic spinning hard drive, an available block can be overwritten regardless of what data was in that block previously; the old data gets overwritten directly. However, the same is not true for flash memory: a flash memory block must be erased explicitly before new data is written to it. And this erase operation is relatively expensive (i.e. slow). If an SSD drive was to erase memory blocks “on demand”, i.e. only when a new file is being written, it would slow down the write performance of the drive significantly.

Therefore, an SSD drive will erase unused memory blocks preemptively, so that the memory will be pre-erased when a new file needs to be written to it. Since the drive has no knowledge of what filesystem exists on it, the drive relies on the operating system to inform it about which memory blocks are no longer used. This is done using the TRIM command: When the operating system deletes a file, in addition to updating the necessary filesystem structures, it also sends a TRIM command to the drive, indicating that the memory blocks occupied by the deleted file can now be considered “stale”, and queued up for erasing.

The SSD drive erases TRIMmed blocks in the background while the drive is idle, transparently to other operations. In effect this means that for any file that’s deleted from an SSD drive, once the drive purges those stale blocks, the actual contents of the file will be wiped permanently from the drive, and will no longer be recoverable.

The above is a slight simplification, since SSD drives also perform wear-leveling which uses rather complex logic involving copying and remapping logical addresses to different physical memory pages, but the general point stands.

Exceptions

There are a few cases when deleted data may be recoverable from an SSD drive:

If TRIM happens to be disabled for some reason. As mentioned above, the TRIM feature is something that is enabled at the level of the operating system. It is usually enabled by default for performance reasons. Nevertheless, most operating systems will let you check whether or not TRIM is enabled, and optionally disable it. For example, in Windows you can run the command fsutil behavior query disabledeletenotify to see if TRIM is currently enabled.
If you’re using an external SSD drive connected over USB. Support for issuing the TRIM command over a USB connection is relatively new, and is not yet supported by all USB controllers and operating systems. If you deleted files from an external SSD drive that’s connected to a USB port, there’s a fair chance that the data might be recoverable.
If you attempt to recover the files immediately after they’re deleted, and the drive provides the contents of stale blocks (which is rare). As mentioned above, the TRIM command puts the deleted memory blocks in a queue of stale blocks, so it’s possible that the SSD drive won’t actually erase them for a short while. The timing of when exactly the TRIMmed blocks are erased is entirely up to the drive itself, and differs by manufacturer. If you search the drive for deleted data sufficiently soon after it’s deleted, and the drive doesn’t return null data for stale blocks, it may still be possible to recover it.
Due to the way that SSD drives perform wear-leveling, it may be possible for stale blocks to get reallocated and copied to different physical positions in the drive, leaving behind the original data in their old locations. Unfortunately this kind of data is generally not accessible using any software tools, including DiskDigger, and can be accessed only by disassembling the drive and reading the physical flash memory chip directly, which is a very expensive procedure done by enterprise-level data recovery labs.

Summary

Despite the above challenges, there’s no harm in trying to use DiskDigger to recover files from your SSD drive, and in certain cases it will be successful. However, if you’ve deleted files from an internal SSD drive, the overall prognosis for recovering them is unfortunately not good.

(This entry was re-posted from my personal blog)

Uncategorized

Recovering data from QIC-80 tapes: another case study

Post author By Dmitry Brant
Post date March 28, 2020

In another of my recent data recovery cases, the patient was a QIC-80 (DC 2000 mini cartridge) tape that was in pretty bad shape. From the outside I could already see that it would need physical repairs, so I opened it up and found a harrowing sight:

The problem with QIC cartridges in general is that they use a tension band to drive the tape spools. If you look at a QIC cartridge, it’s completely enclosed, except for a plastic wheel that sticks out and makes contact with the capstan mechanism when it’s inserted into the tape drive. The flexible tension band is tightened in such a way that it hugs the tape spools and drives them using physical friction.

This tension band is a major point of weakness for these types of tapes, because the lifespan of the band is very much finite. When the tape sits unused for many years, the band can stiffen or lose its friction against the tape spools, which can result in one of two scenarios:

The tension band can break, which would make the cartridge unusable and would require opening the cartridge and replacing the band. This is actually not the worst possible outcome because replacing the band, if done properly, isn’t too disruptive of the tape medium itself, and usually doesn’t result in any data loss.

A much worse scenario is if the tension band becomes weakened over time, such that it no longer grips the spools properly, so that the next time you attempt to read the cartridge, it will spin the spools inconsistently (or cause one of the spools to stall entirely), which will cause the tape to bunch up between the spools, or bend and crease, creating a sort of “tape salad” inside the cartridge, all of which can be catastrophic for the data on the tape. In this kind of case, the cartridge would need to be disassembled and the tape manually rewound onto the spools, being extremely careful to undo any folds or creases (and of course replace the band with a new one, perhaps from a donor tape). This will almost certainly result in loss of data, but depends greatly on the degree to which the tape was unwound and deformed.

Note that the tape drive that is reading the tape is relatively “dumb” with respect to the physical state of the tape. It has no way of knowing if the tension band is broken, or if the tape isn’t wound or tensioned properly, or if what it’s doing is damaging the tape even further. Great care must be taken to examine the integrity of the tape before attempting to read it.

With this cartridge, it’s clear that the tension band has failed (but didn’t break). The tape has obviously bunched up very badly on both spools. Less obviously, the white plastic wheels at the bottom show evidence that the tension band has degraded, with bits of residue from the black band being stuck on the wheels. The fix for this cartridge was to remove the bad tension band, clean the white plastic wheels, respool and tighten the tape, and install a new band from a donor tape. After the procedure was complete, more than 99% of the data was recovered. The tape header was readable, as were the volume tables. Only a few KB of the file contents were lost.

Therefore, when recovering data from very old QIC tapes, it’s probably a good idea to replace the tension band preemptively with a known-good band, to minimize the chance of breakage and damage to the tape. This is why I keep a small stockpile of new(er) tapes from which I can harvest the tension band when needed. At the very least, it’s a good idea to open up the cartridge and examine the band before making any attempts to read data from it.

[note: this post is re-published from my personal blog.]

Uncategorized

Send us all your floppies! They nourish us.

Post author By Dmitry Brant
Post date June 8, 2019

Recently I had another data recovery case that involved a comically large number of floppy disks, as in… more than five hundred (split evenly between 3.5” and 5.25” disks). We’re talking several large USPS boxes packed to the brim with floppies.

Of the numerous 3.5” floppies, only about 10% had one or more bad sectors, and none of them were completely unreadable. The same was true for the 5.25” floppies, even though some of them were physically bent or warped, to the point where I had to cut them open and transplant the disk itself into a new container. Some of the oldest files on these disks dated all the way back to 1986!

The recovery was performed using two older PCs, each of which have both 3.5” and 5.25” internal floppy drives, allowing the reading to be done somewhat in parallel.

There are actually plenty of cheapo floppy drives that connect over USB that can be purchased even now for as little as $15, but these drives are not, I repeat not suitable for recovering data from actual old floppy disks. They must be read by a proper original floppy drive, preferably from the same era as the disks themselves.

Anyway, when floppy disks were in widespread use in the 1980s and 1990s, they weren’t really intended or marketed as a long-term storage solution, but they’re proving to be quite resilient as time goes by. I’m not nearly as optimistic that today’s USB flash drives or SD cards will be readable in 30 years.

To be fair, these old disks have a much lower data density than modern storage media, so it makes sense that they would be more resilient to wear and tear. But still, it’s impressive that even what seems like mediocre-quality floppy disks still hold up to this day.

Despite these excellent outcomes, this still underscores how important it is to recover this data now, rather than waiting any longer and risking these disks developing any more bad sectors. So, let this be a call to action: if you have any old floppies lying around (or old tapes, Zip disks, Jaz disks, or anything else!), contact us for details on how to send them over, and we’ll recover the data from them for a fraction of the cost of other companies.

[note: this post is re-published from my personal blog.]

Blog

Recovered photos looking “cut off” or “half gray”?

Post author By Dmitry Brant
Post date January 22, 2014

Numerous users of DiskDigger have contacted me regarding recovered photos that sometimes appear to be “cut off” somewhere in the middle, with the rest of the photo replaced by a single “color,” or meaningless shapes and colors, like the picture below. What’s the reason for this mysterious phenomenon? The explanation has to do with file fragmentation.

Files stored on your hard drive (or memory card, USB drive, etc.) are organized using a file system, such as FAT or NTFS. These file systems allow for files to be fragmented, meaning that portions of a single file can be scattered across different locations on the drive. For a quick explanation of why files might get fragmented, refer to the figure below.

When your disk is empty, and you start writing files to it, the files will be stored consecutively (“file 1”, “file 2”, “file 3”, etc.), as we might expect. Now, suppose we delete “file 2”, which would leave a “hole” between “file 1” and “file 3.” And now, suppose we write a new file (“file 4”), which happens to be larger than “file 2” was. A portion of “file 4” will be written into the hole left behind by “file 2”, and the rest of the file will be written to the space beyond all the previous files.

In the above figure, “file 4” consists of two fragments. The sizes and locations of the fragments are indexed in the file system structures that appear at the very beginning of the disk.

So, what are the implications of this when a fragmented file is deleted?

As we know, the act of deleting a file doesn’t actually wipe the contents of the file from the disk. Therefore, we can be sure that the actual contents (or fragments) of the file will remain intact, until they’re overwritten by new data, since the deleted file is now treated as empty space, up for grabs.

However, what happens to the file system structures that keep track of the locations and sizes of the fragments? This depends on the file system:

Under the FAT family of file systems, the fragment locations are recorded in the File Allocation Table, which is the namesake of the file system! The bad news is that, when a file is deleted, its File Allocation Table entries are wiped permanently. It’s still possible to locate the beginning of the first fragment of the file, but any additional fragments are considered lost. In DiskDigger terminology, “dig deep” mode (which relies on file system information) will not be able to recover a fragmented file, if the underlying file system was FAT. It will, however, be able to recover the first fragment of a file. And if that file happens to be a .JPG photo, the image will appear to be “cut off” where the fragment ends.

On the other hand, under the NTFS file system, the fragment locations are stored in the Master File Table (MFT) entry associated with the file. When a file is deleted in NTFS, its MFT entry gets marked as “deleted,” but the entry itself is not wiped, so its fragmentation data remains preserved. As long as the MFT entry is not overwritten by a new file, it’s possible to recover the original file in its entirety. In DiskDigger terminology, “dig deep” mode may be able to recover a fragmented file from an NTFS file system, as long as the file’s MFT entry, as well the actual contents, haven’t been overwritten by new data.

And what about “deeper” mode in DiskDigger? Let’s recall that “deeper” mode disregards the file system altogether, and thoroughly scans the entire disk for the presence of files. The problem is that, in the case of .JPG files, DiskDigger can only detect the beginning of the file, since only the beginning has a unique byte signature. So, once again, if the detected .JPG file is fragmented, we’ll only be able to recover the first fragment, since the other fragments don’t have a unique signature.

To summarize, the reason that certain recovered photos may appear “cut off” is that the .JPG file was fragmented, and only the first fragment could be recovered. The only way that DiskDigger can currently recover fragmented files is in “deep” mode, and only if the underlying file system is NTFS. If “deep” mode does not produce the desired results, then the only alternative is to use “deeper” mode, and settle for incomplete or “cut off” images.

It’s also worth mentioning that other data recovery tools similar to DiskDigger have the same limitations. There is ongoing research regarding recovery of fragmented files without referencing the file system, and eventually certain methods of recovering such files may be built into DiskDigger. Stay tuned, as always.

Blog

DiskDigger Pro for Android!

Post author By Dmitry Brant
Post date November 4, 2013

I’m pleased to announce the release of DiskDigger Pro for Android! This new version of DiskDigger is capable of recovering (carving) over 20 different types of files from your Android device’s internal memory, or an external memory card. This includes support for .JPG photos, .MP3 and .WAV audio, .MP4 and .3GP video, raw camera formats, Microsoft Office files (.DOC, .XLS, .PPT), and more!

As with the non-Pro version of DiskDigger for Android, this app requires root privileges on the Android device. The non-Pro version of DiskDigger will remain available (for free!) on the Google Play store, and can still be used for recovering .JPG photos.

So what are you waiting for? Go to the Google Play store on your Android device, and install DiskDigger Pro today!

Blog

The problem with recovering text files

Post author By Dmitry Brant
Post date March 12, 2013

This is a question I get a lot, so I thought I would expand on it more formally: “Why can’t DiskDigger recover plain text (.txt) files in ‘deeper’ mode?”

To begin, let me remind everyone that DiskDigger can recover text files in “deep” mode, where it scans the structure of the file system and recovers files based on clues provided by the file system. This makes “deep” mode capable of recovering any type of file, including text files, while only being able to recover these files from a healthy file system.

However, in “deeper” mode, things are very different. Since DiskDigger no longer relies on the file system to parse the structure of files on the disk, it can only detect files based on byte sequences known to exist in certain types of files. For example, all PNG image files begin with the byte sequence 89 50 4E 47. Therefore, DiskDigger can look at every sector of the disk, and if it begins with this sequence of bytes (files must be sector-aligned), it knows that there’s a PNG file at that location.

The same is true for many other types of files, like .JPG, .DOC, .MOV, etc. It’s also true for files that are built from text, but have a consistent structure, such as .HTML, .XML, .RTF, etc.

So now we come to the problem with pure text files. Unlike other types of files, text files do not contain any identifiable sequence of bytes. They only contain… well… text! There’s no underlying binary structure.

This makes it nearly impossible for DiskDigger to “pick out” a text file from all the other random content on a disk.

Despite all this, there are a few remote possibilities for recovering text files which are an active area of development in DiskDigger. None of these are perfect, but they may eventually lead to a solution for recovering some text files:

Some text files may be encoded in Unicode (specifically UTF-16). In this case, the text file will have a starting byte signature, which is either FE FF or FF FE. Unfortunately this signature is far too short to meaningfully identify UTF-16 files, and will produce too many false positives.
Since many text files are encoded in ASCII or UTF-8, and written in English, we can expect them to contain only characters between 0x20 and 0x7F (along with \n and \t). We can then perform a statistical analysis on each sector of data, and if it contains mostly characters within our desired range, we can consider it to be part of a text file. There are several problems with this approach, however: we won’t be able to determine the size of the detected text file, and we won’t be able to tell where one file ends and another begins (if there are two text files next to each other on the disk). Also, since this is a statistical method, it will surely produce false positives, as well.
Some text files do have some semblance of structure in them, in the sense that they have an identifying signature, but not in a consistent location. For example, most C and C++ source files have the word “#include” somewhere near the beginning. By searching an entire sector for this kind of signature (independent of offset), we can be somewhat certain about the presence of a particular file. This kind of functionality is actually already built into DiskDigger’s custom heuristics feature. This method, however, still has the problem of not being able to detect the size of the recoverable file.

As discussed above, it’s generally not feasible to recover plain text files, because they have no discernible binary structure.

It is, however, possible to recover a text file using custom heuristics, as long as you know an exact sequence of letters that is certain to appear near the beginning of the file. I will write a short tutorial on performing this task in a future article. Stay tuned!

Blog

Recovering QIC-150 tapes: a case study

Post author By Dmitry Brant
Post date March 12, 2013
4 Comments on Recovering QIC-150 tapes: a case study

A friend of mine recently found something very intriguing: several backup tapes from his old Amiga 500 computer from around 1995. Since he no longer has the original Amiga machine, he no longer has a way of reading the tapes, and thus no way of rediscovering the old documents, letters, or any other memories that were lost when the old computer was thrown away.

Without hesitation, I offered to help recover the data from the tapes, not only for the challenge of it, but also because I have never dealt with Amiga computers or Amiga-formatted media, and this would be a great opportunity to familiarize myself with a significant part of computing history, even if it has already come and gone. This is a brief chronicle of the steps I took to recover the data, just in case someone else in the future (including myself) needs to perform a similar task.

One of the most pleasurable aspects of this experience was how easily everything came together, owing mostly to the abundance of information on the web regarding every step of these kinds of processes. Hopefully the information in this article will contribute to that abundance.

The tapes were QIC-150 cartridges (specifically Sony D6150). After rummaging a bit through my attic, I realized that I actually own a tape drive that’s capable of reading QIC-150 tapes. The drive is an Archive Viper 2150S, which uses a SCSI interface:

Since the drive is SCSI, I would need a SCSI host adapter card to interface with it. Luckily, after rummaging a bit more, I found an Adaptec AHA 2940UW adapter which looked like it would be perfect for the job:

I installed the adapter into an old PC that “still” has a PCI bus (how far we’ve come!), connected the drive to it with a 50-pin SCSI ribbon cable, and put a terminator on the end of the cable (I could have also used 8-pin SIP resistors for which the drive has sockets). The jumpers on the drive were already set to have a SCSI ID of 0, so I didn’t have to make any physical changes to the drive or the adapter.

I booted up the PC, and was happy to see that the Adaptec card was correctly reporting the Viper 2150S as being connected with an ID of 0. So far, so good!

The computer booted into Windows XP, and at this point I encountered one of the very few hiccups in the whole process: Windows XP does not have a driver for the Viper 2150S drive. Apparently Microsoft discontinued support for it after Windows 2000. That was perfectly fine, since my next instinct was to boot into Linux (I simply used an Ubuntu 12.04 live CD).

Within Linux, the tape drive worked perfectly. I inserted the first tape, and typed the command to rewind the tape to the beginning:

$ sudo mt -f /dev/nst0 rewind

The drive obeyed without any errors! So then, I decided to go for all or nothing: I issued the command to simply dump the entire contents of the tape to a binary file:

$ sudo dd if=/dev/nst0 of=tape1.bin

The drive started whirring, and I watched in amazement as twenty-year-old technology was working flawlessly in the world of the future. After a few minutes, the data dump was complete, and I had a pristine image of the data from the tape. I repeated the same process for the other tapes, which also had zero issues or bad blocks. Three cheers for Sony for developing such a sturdy medium for storage, and to the owner of the tapes for preserving them so well.

After reading the binary images of the tapes came the next hurdle: How is the data formatted? Tapes do not have a “file system” like hard drives do, so the data on the tape is entirely application-specific. In order to make sense of the data, we would need to know the precise application that was used to write it!

Here’s what we know about the data from the tapes:

It’s from an Amiga system
It’s from around 1995

The owner of the tapes did not remember which software he used to make the backups, but we could assume that it was probably a “common” backup application at the time. After some searching around the web, it became clear that the most common backup tools for Amiga at that time were Ami-Back, Quarterback, and Diavolo.

The next step was to set up an emulated Amiga environment, so that we could run the original Amiga software to restore the backups.

The de facto solution for Amiga emulation is WinUAE, which is a very impressive and virtually feature-complete emulator. In order to work properly, WinUAE requires a ROM image (referred to as “Kickstart” in the Amiga world). After that, it can run bootable Floppy images for Amiga, or a bootable Amiga hard drive (with AmigaOS loaded on it) that can be mapped to any Windows folder. (the ROM image and the AmigaOS files are the only semi-difficult things to obtain; beyond the scope of this article)

I found an extremely helpful step-by-step guide for installing AmigaOS 3.9 under WinUAE. To my surprise, I also found a vibrant and thriving community of Amiga users who are willing to help find and share old software.

In no time at all, I had a fully-operational AmigaOS system, ready to attempt some trial-and-error methods of restoring the backup files.

My plan was the following: I would install all three of the most common backup tools for Amiga (Ami-Back, Quarterback, and Diavolo), and use each one to make a test backup. I would then compare the binary format of the test backups to the tape images. With any luck, one of the binary formats would match up.

And, lo and behold, the backup from Ami-Back was a match! So now, knowing that Ami-Back was the correct utility, I mounted one of the tape images as a hard drive in the emulator, and used Ami-Back to “restore” from it. It worked like a charm. The files poured out of the backup image, as I watched, yet again, in amazement.

It was a complete backup of the entire Amiga system from 1995. Since I had already set up the emulator correctly, I was able to make it boot directly from the restored system image. In effect, I was able to see the screen of the computer, exactly as it would have been seen nearly twenty years ago:

Blog

DiskDigger 1.5.5.1507 released

Post author By Dmitry Brant
Post date March 10, 2013

The latest version of DiskDigger is now available for download! Go to the DiskDigger website to check out the new features and download the updated program.

Blog

Welcome to the new site!

Post author By Dmitry Brant
Post date April 24, 2011

Welcome to the new website of Defiant Technologies, LLC! This site will serve as a medium for us to share news about the company, new services we’re providing, and new products we’re developing. Feel free to look around!