A widely used tool for mirroring a site is GNU wget. This article on FOSSwire discusses how to use wget to create a site mirror. Be aware, though, that wget doesn’t do anything but grab files. If your site has dynamic content or if it depends on files that aren’t downloadable, wget won’t help you.

Another tool is HTTrack. Unlike wget, it has a GUI. It’s currently available only for Windows, Linux, and Unix. Like wget, it can only grab publicly available files.

Search engines don’t always deal well with mirrors. If they detect a mirror site, they’ll often list just one version, and they may guess wrong about which one is the primary. This actually happened with the Filk Book Index; for a while, Google and other search sites were listing the index on freemars.org but not the one on massfilc.org. The solution for this is for the mirror site to create a file called robots.txt at the top level of the site directory, with the following content:

User-agent: *
Disallow: /

Be careful not to put that on your primary site, though, or search engines won’t list you at all! wget works best if your primary site doesn’t have a robots.txt at all, since by default it respects its restrictions.

(In case you aren’t familiar with robots.txt: it’s a file syntax which search engines use by convention to determine which files on a site they shouldn’t index. It doesn’t actually prevent any access, so it’s not a security measure, but it’s respected by legitimate web crawlers. Learn more about it here.

What about the increasingly common sites with dynamic content? You can mirror them, but it’s harder. You’ll have to send your collaborator a copy of all your files and make sure that any needed software is on the mirror site. If it depends on a database that’s regularly updated, you may be able to give the mirror site access to it. Of course, if the database goes down all the mirrors go down with it.

A mirror actually doesn’t have to be publicly visible, as long as it can be made visible on short notice. You could, for example, put the mirror in a directory which isn’t available to browsers, and run a periodic script that makes the mirror available if the primary stops being available for an extended period of time. Strictly private mirrors can be useful too, if making them available quickly isn’t an issue; they can prevent content from being lost.

“All flesh is like grass,” wrote the Apostle Peter, “but the word of the Lord remains forever.” It’s certainly true that words can outlast people. Will yours, if you’ve put them online? Do you always want them to? If you do nothing, Murphy’s Law may prevail. The flames you wrote on Usenet when you were young will survive, but the writing you value most may go down the digital drain.

It’s likely you have resources on many different sites. You might have a blog, a website, and a social networking account, and probably more than one of some of these. If it’s something you’re paying for, it could disappear when the payments stop. If it’s a free site, it might be terminated for inactivity, In either case, there might be material — restricted posts, private data, infrastructure — that no one can capture by looking at your site.

Some social networking and blogging sites allow family members or friends to “memorialize” an account. I’ve successfully requested this on LiveJournal for two friends who died late last year. How this is handled varies from one site to another. LiveJournal retains all posts. Facebook’s policy is to delete all status updates. Facebook’s approach maximizes privacy but could also wipe out information about someone’s last days that isn’t available anywhere else. Yahoo goes even further, giving your heirs no access to your account except the right to request its deletion. (It’s a little morbid to say “you,” but I have to use some pronoun, and it’s your own legacy you have to be most concerned about.)

In the stress and confusion following your death, your information might be lost. Perhaps no one will know what accounts you have or what their passwords are. (Normally the latter is a good thing, but not in this case.) The best plan may be to have a copy of everything that’s valuable on your own computer and to make sure someone in your family knows how to get at it. This is easier than scrambling around multiple websites with multiple accounts. If you do keep information online which you want to survive you, keep a list of accounts and passwords in a secure place, and make sure someone knows it exists.

It may help to put provisions in your will directing the disposal of your important online assets. I’ve seen a claim that Facebook will download the complete contents of a deceased user’s account to an heir, if you’ve specifically requested it. Such provisions may let your heirs override sites’ default policies. Not being a lawyer, I won’t offer any suggestions on how to phrase such directions, and I’m guessing a lot of lawyers don’t know either.

If you’ve written and uploaded stories, poems, or songs, then you might want to take steps to make sure they can legally stay online. A provision in your will to assign your copyrights could help, and — painful as it might be to realize this — your family members aren’t necessarily the best people to assign them to, especially if your works have only literary and not monetary value. Maybe your kids don’t really care for the beautiful fanfic you wrote. If you assign the copyright to someone who does care, it’s less likely to vanish into a legal black hole.

It’s a difficult area to think about and a difficult one to make the right decisions in, but some planning can make a difference.

Links:

• The Digital Beyond. A whole website about digital inheritance.
• Facebook’s “If I Die” App Should Remind Us That We Each Need a Digital Death Plan (Justia)
• The Big Digital Sleep (Library of Congress)
• Digital Death and Digital Afterlife: Serious Business (Forbes)

Just figuring out what format you have is confusing. There are two things you have to know: the format of the file as a whole, called the “container,” and the way the bits represent the video, called the “encoding.” These are largely independent of each other, and the specifications for each can have multiple options. The same format may be referred to by different names, and different formats may be called by the same name.

Usually you create a video from a camera, and it probably doesn’t give you a lot of format options. If you process it with a video editor, you have more choices about the final format. The file suffix tells you what the container format is supposed to be but not what encoding was used. If it’s .MOV, you have a QuickTime container. If it’s .MP4, you have an MP4 container — which is not synonymous with MPEG-4, but rather with MPEG-4 Part 14. Both are MPEG-4 compliant but not at all compatible with each other.

It’s common to refer incorrectly to other MPEG-4 container files, including audio-only files, as MP4. If it’s not a Part 14 container, it shouldn’t be called MP4. On the other hand, its being a legitimate MP4 file tells you nothing about what encoding it uses, so not all “MP4″ files are compatible with each other; likewise for QuickTime files.

Videos produced by current cameras and software usually will use the H.264 encoding, aka MPEG-4 Part 10. You may also run into MPEG-4 Part 2, which is based on H.263. If you have a strong preference for open-source, you may want to go with the Theora codec. The win is available source code and (hopefully) a lack of patent encumbrances, but the risk is that less software supports it. Preservation is always a matter of placing bets. If you use Theora, it should be in an Ogg container, not an MP4 container; the latter combination is technically MPEG-4 compliant, as a “private stream,” but may not be supported in the long term.

An older container format, Audio Video Interleave or AVI, still has strong support. It dates all the way back to Windows 3.1. Its Full Frame encoding option lets you store uncompressed video.

Microsoft’s current entry is Advanced Systems Format (ASF), often in combination with Windows Media Video (WMV) encoding. It’s widely supported but tends to be Windows-specific, so it may not be the best choice for long-term preservation.

Most video encodings are compressed, since they take a lot of space even by modern standards, and usually the compression is lossy (i.e., it isn’t possible to recover the original data without some loss of accuracy). There are ways to get uncompressed or lossless compressed encoding, but here we’re getting into esoteric areas which I’d best not touch.

The video encoding isn’t the whole story. An encoding such as H.264 is only a video encoding, and even if you’re a silent movie fan like me, you probably like sound in a lot of your movies. The audio encoding that goes with H.264 is usually MPEG-4 Part 3 Advanced Audio Encoding, known for short as AAC, but this isn’t required. If it’s something else you could have preservation issues.

As I said, it’s a mess. I’m far from an expert in this area, but this article should give you an idea of the issues to look for.

Suggestions: Current advice varies a lot. Popular options today include a QuickTime (.MOV) or MPEG-4 Part 14 (.MP4) container with H.264 video and AAC audio if you want to go with software popularity, or Ogg with Theora and Vorbis if you value openness more. The older AVI is hardly dead. Pay attention to the encoding, not just the container format. Stay tuned for future developments and be prepared to migrate to new formats.

Links:

• Whither digital video preservation? (Library of Congress)
• Sustainability of digital formats: Format descriptions for moving images (Library of Congress) This hasn’t been kept up to date very well, and the exclusion of Theora is conspicuous, but it lists a lot of formats.
• Video File Formats: What and when to use.
• Preserving digital video (Disruptive Library Technology Jester)

Here I’m not addressing the moral or legal propriety of the action, but the implications for file preservation in the fact that it could and still can be done. I’ve heard that Amazon has renounced the policy of retroactive withdrawal, but the technical ability still exists. Amazon uses it to implement its money-back policy; you can “return” a book within a certain time period, and Amazon will refund your money and delete the book.

The underlying problem is Digital Rights Management, or DRM. This refers to any technology that impedes the making or use of unauthorized copies of a digital document. If you buy a book which is under DRM, you haven’t really bought it; you’ve leased it for an indeterminate period. If you want to move a Kindle book to a different computer or reader, you can’t just copy the file over; you have to re-download it. It’s been reported that Amazon may put an undocumented limitation on the number of times you can download a book, or perhaps (as mentioned in the comments) an undocumented limit on the number of registered devices it can be on. According to mobileread.com, Amazon says that “Most books come with a maximum of six licenses, but their are some books that may have less. The amount of licenses a book comes with is determined by the author or publisher who owns the right to the content.”

The long-term risks can be worse. There is no guarantee any business will be around next year, and no guarantee that it will continue to support its products. Amazon isn’t likely to vanish quickly, but someday the Kindle format will go the way of Kodachrome film. When that happens, all the DRM’ed books you’ve bought for it will be a digital pile of ashes.

A company can pull DRM just because it feels like it. In 2007, Major League Baseball dropped DRM support for game videos that people had “purchased.” No refund, just poof. Similarly, when Google dropped Google Video, the content which people thought they had bought was gone. Google issued refunds, but only after coming under public pressure. Major League Baseball also backed down, though problems remain. It’s likely that less-publicized instances with other vendors have left customers permanently stuck.

DRM is an attempt to solve a real problem, the unauthorized distribution of digital copies of works with no compensation to their creators. I have no objection to its use with rental or single-view distribution of materials. But any document under DRM that you “purchase” will, sooner or later, die. If a book was published only under DRM, it’s gone, unless someone has hacked out an unprotected copy of the content. Under the Digital Millennium Copyright Act, breaking DRM protection is usually illegal, even if it’s to make a personal backup copy of something you’ve paid for.

The books on my shelves will last a good long time if I take good care of them. So will e-books that I download, if they’re in a standard format and I take care to back them up and convert them to a newer format when needed. But books under DRM are under the curse of death.

Suggestions: Books and other documents under DRM are like beer. You can’t buy them; you can only rent them. Don’t “buy” any content under DRM unless you’re willing to consider it a throwaway copy.

Like any good, cheap thing, online storage has its drawbacks. One concern is security. Someone could intercept your files in transit, or break into the storage site, and grab them. If you’ve got serious confidential information on them, such as credit cards, Social Security numbers, and passwords, this can be very bad. In some places, such as Massachusetts, it’s illegal to store what’s called “High Risk Confidential Information” or HRCI offsite without taking certain precautions. On the other hand, in some cases, such as medical services, you may be required to have offsite backup and to make sure it’s secure.

You can get some security by encrypting your files. Be sure that the encryption key is a good one and that it’s available to those who may need it, possibly years later, and not available to anyone else, including the backup site.

Online storage may go away unexpectedly. Omnidrive, a company based in Australia, was once highly regarded for its Web-based storage service, but it went away in 2008 without warning, along with all the files stored on it. It hasn’t been the only one.

When you start using an online backup service, be sure to test its restoration features early on. After a disaster happens isn’t the time to discover you’ve been copying files into a black hole.

Don’t look at a site’s terms of service. These exist to protect the site owner and guarantee absolutely nothing. The terms of service from Iron Mountain, one of the most reputable storage companies, say:

Neither Iron Mountain, any of its affiliates, directors, officers and employees, nor any other party involved in creating, producing or delivering the Site is liable for any direct, incidental, consequential, indirect or punitive damages arising out of your access to, or use of, the Site or the operation of the Site or failure of the Site to operate. In no event shall Iron Mountain be liable for any direct, indirect, special, punitive, incidental, exemplary or consequential, damages or any damages whatsoever, even if Iron Mountain has been previously advised of the possibility of such damages, whether in an action in contract, negligence, or any other theory, arising out of or in connection with the use, inability to use or performance of the information, services, products and materials available from this Site. These limitations shall apply notwithstanding any failure of essential purpose of any limited remedy.

What you need to look at, if you’re paying for the service, is the Service Level Agreement (SLA). These will, for the better sites, promise you reimbursement if service fails to live up to a specified standard. Not all sites post theirs online, but you should make sure to see it before committing to any money. At best you’ll just get part or all of your money back, which can be scant consolation for losing your files, but it’s an incentive for the company to meet its obligations. If you’re using a free service, then of course there’s no guarantee of any kind.

Internet bandwidth is a factor to consider. If you have lots of frequently changing files to back up, you may need to get a faster connection to keep up.

As with a local drive, you’re best off automating your backup so you don’t have to remember to do it. Most services offer software that will let you do this. Make sure the software offers reasonable security and doesn’t send your password in cleartext. Make sure you understand what it’s backing up: Some services back up your whole drive by default, possibly including stuff you don’t want anyone else to have a copy of, even encrypted. Stuff you keep for … artistic reasons.

Don’t just save and forget. Check occasionally on news about your backup service, so you don’t get caught by surprise if it gets sloppy, is acquired, or goes out of business.

Suggestions: If you take suitable precautions and make Internet-accessible services a part of your backup strategy, your data can be safer than before. But don’t make the “cloud” your primary backup plan if you don’t want your files to go up in smoke. Use more down-to-earth techniques for long-term storage.

Personal note: On Sunday, February 19, I’ll be on a panel at the Boskone science fiction convention on “Digital Estate — Virtual Property OR On the Internet, Nobody Knows You’re Dead.”

Links:

• Choosing Cloud Backup For PCs
• Disaster-Proof Your Data with Online Backup
• Amazon Simple Storage Service
• Online backup service reviews

If used carelessly, though, it has its risks for long-term preservation. A PDF file isn’t necessarily self-contained. It might depend on external fonts or even require whole files of content. To avoid the risk of dependencies that might cause future problems, you can use PDF/A, a restricted subset of PDF. Good software for generating PDF usually includes a PDF/A option.

On Macintosh OS/X doing this is a little weird. You start by selecting “Print” from your application’s File menu. There will be a “PDF” button in the Print dialog. Clicking on it brings up a menu. Choose not “Save as PDF…” but “Save as Adobe PDF.” This launches an application to save the PDF. You’ll see a dialog like the following:

In the “Adobe PDF settings” select one of the PDF/A options.

If you have Acrobat, conversion to PDF/A is much simpler.

Correction (29-Jan-2012): If you don’t have Acrobat installed on your computer, you don’t get the “Save as Adobe PDF” option. Sorry about that.

PDF/A documents are even harder to edit than regular PDF’s, so you should defer converting a document to PDF/A if you’re still tweaking it.

There are two versions of PDF/A, called PDF/A-1 and PDF/A-2. A-1 is based on PDF 1.4, and A-2 is based on 1.7. PDF 1.7 is fully backward compatible with 1.4, so you’re safe using either. There are two compliance levels, a and b, with a being the stricter. From a preservation standpoint, there isn’t much difference. Converting to Level b may be easier, since Level a requires you, or the generating software, to produce a tagged structure. The structure helps to make the document self-explanatory but may not be necessary.

There are quite a few restrictions which a file must satisfy to conform to PDF/A, including these:

• All fonts must be embedded in the document, and there are some restrictions on font implementation. Conversion software will generally refuse to put fonts flagged as non-embeddable into PDF/A.
• Color profiles must be used to guarantee device-independent color.
• The document may not be encrypted.
• There may not be audio, video, or JavaScript.
• XMP metadata must be included.
• Forms are allowed with restrictions.

Clearly there are legitimate things to do in PDF that can’t be done in PDF/A. Audio and video can’t be included. The requirements for fonts and profiles can make PDF/A files bigger than they’d otherwise be. But if it’s important to give files the best chance for long-term readability and the restrictions don’t impact necessary content, PDF/A is a good choice.

Suggestions: Use PDF/A for documents that you expect to be needed for a long time. If you’re using PDF as a general-purpose container, though, it may not be a good choice.

Useful links:

• PDF/A-1, PDF for Long-term Preservation, Use of PDF 1.4 (Library of Congress)
• PDF/A Competence Center
• PDF/A FAQ (PDF)
• Another PDF/A FAQ

If you’re stuck in such a situation, there are tools such as HTTrack for downloading whole websites. That will at least give you the raw material for building a new site. But loss prevention is always better than piecemeal recovery. Many services have backup tools. For example, on WordPress you can export your site as XML. It may take a little work to move that to anything but another WordPress site, but you have all the material. The site maintainer for your organization should do regular backups or exports if that’s an available option. Don’t say you’re afraid they’ll walk out on you, of course; just say “in case you get sick” or whatever the best excuse is for your cultural group.

If the service allows more than one administrator, take advantage of that opportunity. Oh, the two admins shouldn’t be a married couple. Better yet, have the site administered by a group-owned account, with the password securely stored in more than one place. Accounts are usually tied to email addresses, so create an account such as webmeister@your_name_here.org and register the site with that address.

Sometimes people donate their own server space. This can be an attractive offer, since it costs nothing and makes you completely independent of any business that might vanish, change its terms of service, or arbitrarily cancel your account. But people can vanish too. (I’m speaking as someone who’s had two friends die in the past two months.) Weigh the risks, and keep a backup that will stay available.

Suggestions: Don’t keep all your eggs in one basket. Whatever approach your organization takes, make sure it isn’t critically dependent on one person.

You really should start by understanding how Time Machine stores files. If you look at a Time Machine backup volume that’s been in use for a while, you’ll see a directory called Backups.backupdb. This is the root directory for all your backups. Under it will be a directory named for your computer. If you’re backing up more than one computer to the same drive, they’ll all be listed there. Under that will be some number of directories whose names look like timestamps, and a directory alias called “Latest.” You should see directories for every hour your machine has been on in the recent past, dwindling down to dailies when you go back a day or two, and weeklies when you look at directories more than a month old. There may be gaps, depending on how much of the time your computer is turned on.

If you look under any of these timestamps, you’ll see what appears to be a complete copy of the contents of your drive. Add all of these copies together, and you may find it contains many more gigabytes of data than your whole drive! How does it do this? The answer is incremental backup and the Unix “hard link.” When it does a backup, only files that have been added or changed since the last backup are copied; the rest are just directory links to older backup directories. Unlike an alias, a hard link is a first-class citizen of the file system; for all practical purposes, the file is there, and it’s also where it used to be. A hard link, like a Tardis, lets your files be in two or more places at once. The nice result of this is that if you want to get a particular file or folder back, you just have to find it on the backup volume and copy it.

Older versions of your files, as well as deleted files, are retained in older backups. Eventually your drive will fill up, and TM will delete the oldest backups, so don’t count on it to keep anything forever.

As far as I can tell, Apple doesn’t say what happens if you shut down your computer while a backup is in progress. This help item says, for OS X 10.7, that if a backup is accidentally interrupted, “Time Machine resumes the backup where it stopped.” A test shows that if I shut down during a backup, a file with a name ending in “.inProgress” is left, and a backup directory appears only after the backup is complete. If Time Machine is in the middle of a long backup and need to leave your computer, you’re probably better off letting it finish rather than making it start over.

Knowing how Time Machine works provides some clues about how to use it most effectively. It automatically excludes some files, such as caches, from backup, and you can tell it to exclude more. If your backup drive is less secure than your computer, you might want to exclude files that have serious confidential information. Time Machine handles large database files poorly, since all it knows how to do is make a new copy of a file that’s been changed. They could be another candidate for an alternative backup strategy. But make sure you have some reliable backup; those database and confidential files are probably important, and losing them completely isn’t good security!

With some work, you can have Time Machine back up a database monthly (or however often you like) even if you change it every day. Exclude the database file from backup, and set up an automated task such as a cron job to copy it once a month to a directory that isn’t excluded. Time Machine will back up the copy only when it changes.

Avoid the temptation to delete files from Time Machine. because of the way directories are linked together, you might either lose more than you think or not really delete what you think you’re deleting.

Multiple backups are better than one, and a drive that isn’t usually connected to your computer is safer than one that’s always connected.

These are just a few tips for using Time Machine effectively. To learn lots more, read the articles in the links below.

Suggestions: If you have a Mac, take advantage of Time Machine to automate your backups, but use it intelligently.

Useful links:

• Mac 101: Time Machine (Apple)
• Time Machine: Frequently asked questions
• Time Machine problems (Apple, OS X 10.7)

“Open PodBay.doors, Hal.”

“I’m sorry, Dave. I’m afraid I can’t find an application to do that.”

For several years I used a Macintosh page layout application called “Ready, Set, Go!” to make songbooks for filk music conventions. Somewhere along the way I upgraded my system and discovered that RSG would no longer run. All the files I’d edited with it were so much digital junk.

This is a user’s nightmare: A change in the computer or operating system makes an application obsolete, and there’s no way to recover the files that you made with it. The best way to avoid this situation is prevention. If you care about the long-term usability of your files, either avoid creating them in an obscure proprietary format or regularly back them up to a more robust format. The tests are whether there’s more than one application (preferably several) that can deal with the format, and whether the format specification is publicly available and standardized. This is often put as “proprietary” vs. “open,” but that can be misleading; some open formats are proprietary in the sense that a particular business owns them, but open in that the specifications are published and anyone is free to implement them. (An example of this is the TIFF image format, which is specified by an Adobe document but widely implemented.) On the flip side, “open” formats may not be as open as they seem if they allow for vendor-specific data which can be critical in practice to proper rendering. (An example of this is Microsoft Office Open XML.)

If your application creates files in an obscure proprietary format, be sure to export them to a more portable format, and make sure they’re all exported when you’re about to do a system upgrade. If it doesn’t have any export capability, look for add-ons that might allow it. If there aren’t any, you might seriously consider changing to a different application before your potential losses get too big. Sometimes, though, the big-name applications cause just as much trouble. If you have files created by an old version of an application, the current release may not be able to open them.

Outputting files as PDF (except for audio and video, of course) can provide a useful safety net. Converting them back to an editable form may be difficult, but at least you have the content in a usable form. For maximum safety, specify PDF/A when you do this.

But what if you’re already stuck in a blind alley? You upgraded your computer and now you can’t open files that you desperately need. The first thing to try is right-clicking on the file icon to see if it offers any applications that can open a file. If the first one you try doesn’t work or does a poor job, try any others. If you manage to open the file that way, be sure to save it in a more current format while you can.

If that doesn’t work, there can be other solutions, generally requiring effort or money. You can get a computer with an older version of the operating system and install your old application there, if you can still find the installation disk. The fact that you probably don’t need the latest hardware to do this may make this a relatively cheap, though still effort-intensive, approach.

You can pay an expert to convert your files for you. Consulting outfits such as Geek Squad offer such services. I make no recommendations here; if you go this route, you need to look for someone who’s reputable and has a record of results, and even then you may have a situation too obscure for them to handle. It’s doubtless easier if you want to recover files that just went obsolete on your computer than if you ask someone to get back files you haven’t been able to read for five years.

Sometimes partial recovery is possible. On several occasions, I’ve salvaged old word processing files by opening them in a text editor and fishing the readable text out of the markup. If the formatting isn’t critical, this can be very effective. This isn’t likely to work with image or sound files, though.

Here I should put in a special warning about Mac OS X Lion. It no longer supports Rosetta, the compatibility software that allows applications written for the older Power PC processor, to run on Intel machines. This means that any old Power PC applications you might have will just stop working if you install Lion. The simplest solution for now is not to use Lion; as far as I’m concerned, it offers no advantages and makes several things worse. But eventually you’ll want to get software that won’t run on anything older, so this just postpones the issue for a year or two. That year or two can give you enough time to make sure you aren’t going to lose anything when you yield to the inevitable.

Suggestions: Before upgrading your operating system or an important application, think about what files might be at risk. If in doubt, export or convert them to a safer format. If you have files that were created by an old version, update them to the current version. Prevention is much easier than curing.

Links of interest:

• Getting ready for Lion: Converting old files
• Office Migration Planning Manager overview for Office 2010 (Microsoft TechNet)
• “It’s Dead, Jim”: Rescuing an obsolete file (The Signal, Library of Congress)