Contents, Abstract ... and Domesday Reloaded

The BBC Domesday Project was a pair of interactive videodiscs made by the BBC in London to celebrate the 900th anniversary of the original Domesday Book and published in November 1986. It was one of the major interactive projects of its time, and it was undertaken on a scale not seen since. This document is adapted from material written for publication in Japan in 1988 and it sets out some of the background. Domesday may have been a marketing failure but those of us who made it see it as a triumph of interactive knowledge engineering. Over a million people took part in one way or another and it is said that it would take you over seven years to look at everything on the discs. The project was held up to be an example of digital obsolescence (even though the audio-visual content is analogue) and an effort was made to preserve the content, although making it widely accessible was not possible, partly due to questions over copyright, until the 25th anniversary, when the BBC felt able to republish the Community Disc material online. The fate of the National Disc material is, as yet, still undecided while (you guessed it) copyright issues are considered. Despite such potential problems there were (and probably still are) people working away both inside and outside the BBC to republish, port and build on the original project work ... and even rip high quality video from the original videodiscs.

Domesday Reloaded 2011 and beyond

In the meantime, and over a period of several years, a BBC team worked towards the 25th anniversary of the project, under the title Domesday Reloaded. This was launched across the BBC, notably on Radio 4 and BBC Local Radio. A few of the team members gathered in a studio in Bush House in March to talk about the production back in the mid-1980s in the company of Michael Wood. This was for an Archive on 4 programme which was broadcast on May 14th. The contents of the Community Disc (photographs and text from members of the public, plus vintage OS maps to go with them) were put onto the Domesday Reloaded web site with a new interface and brand new maps to compare. There was a 'call to action' to identify, geotag and update the photographs (and hopefully give them their long-missing photographer credits) which closed at the end of October 2011. The material then passes on to the UK National Archives for 'safe keeping' alongside their archive of UK government web sites.

Early in December 2011, a touch-table version of some of the project (Community maps and text plus 1980s news clips) was installed at the National Computing Museum in Bletchley Park [BBC news story including video]. This aims to be a permanent exhibit alongside original Domesday Project systems and a room full of BBC Micros.

I presented (or rather croaked my way through) the 2011 Jane Mercer Memorial lecture for FOCAL International and the Royal Television Society. The presentation was available on the FOCAL International web site but is, at the moment, off-line.

In 2019 I was pointed to a couple of excellent papers by Richard Tapper, who was one of the project specialists, which look at the educational and data management aspects of the project. They were written around the time of the project's publication in 1987.

• The Domesday Project: an educational review
• Building the Domesday Database: lessons for integrated database development

Some coverage of Domesday Reloaded...

• Domesday Community Disc maps and text using a Google Maps interface
• Domesday Reloaded web site (Via UK National Archives)
• BBC Domesday Reloaded Blog
• BBC Press release
• BBC News story on launch
• BBC News story on content re miners' strike
• Podcast interview with Peter Armstrong (from Wired ... about 16:40 into the file)
• Video compilation from May 5th pre-launch event at BBC TV Centre

• Bonus item: New online version of the original Domesday Book
• Bonus item: Acorn User article (1985) on schools Domesday research
• Bonus item: 1982 New Scientist piece by Barry fox on interactive videodiscs (on Google)
• Bonus item: BBC Domesday: The Social Construction of Britain (by Alan Macfarlane from the fall 1990 issue of Visual Anthropology Review and giving a fascinating look behind some editorial processes of the project)

Andy Finney (project co-founder and member of the 1984-86 production and 2011 touch-table version teams): 1996-2021

---

Domesday and the Norman Conquest of England

1986 was the 900th anniversary of the Domesday Book. This document, regarded as the first great work of a bureaucratic state, was commissioned by King William I of England (known as 'The Conqueror', amongst other things) to provide a record of the land he had conquered 20 years before in 1066 ... the only historical date every English person knows. The original book was hand-written by, probably, a single monk. It still exists (actually there is more than one volume) and is kept at the UK's National Archives in Kew (near where the famous Gardens are) on the edge of central London. The book set out to take a record of all the lands in the kingdom and to record who held it for the King. The irony is that few of his subjects could read ... but incompatibility has always been an issue in information technology.

In 1983, a BBC Television producer named Peter Armstrong wondered if it would be possible to harness the Domesday philosophy to modern Britain. With the large user base of microcomputers in British schools (helped by a government subsidy) it was feasible to ask schools around the UK to survey their areas to produce a database of how Britain looked to the British in 1986. Combine this with central statistical, written and visual information and you have the basic structure of the 1986 Domesday Project.

Technical considerations for a Modern Domesday

This modern Domesday would need to contain text, photographs, data and a controlling computer program to bind it all together. It was decided to use maps to provide an index to the information and photographs gathered by the schools around the country. A national photographic competition was launched to produce thousands more photographs showing life in Britain in the '80s. The missing part of the equation was a delivery system that could handle large amounts of information and pictures.

In a sense interactive video came looking for Domesday. IV was in use in small scale projects but was not seeming to reach its full potential. The recent arrival of compact disc audio was making more and more people aware of optical disc technology but CD-ROM was not really an option in 1983.

The target market for the Domesday Project was seen to be in education and in libraries. Both outlets would prefer a single-item software package rather than the more usual combination at the time of floppy disc and video disc. It was impracticable to consider using CD-ROM alone, because it had, and still has, a limited capacity for pictures and, in any case, we are talking about a time before High Sierra (the original CD-ROM filing system), let alone ISO 9660 and JPEG. Also, CD-ROM was aimed at business users with IBM-PC clones whereas schools had BBC Micros (8 bit 6502 with 32K of ROM and 32K RAM), Sinclair Spectrums or Research Machines computers and sixteen colours was just about the best you could get.

In any case, using floppy discs with a LASER videodisc were out of the question because of the data storage needed and a CD-ROM drive with a LASER player was very much an unknown quantity in those days before High Sierra.

The initial plan was to put full-frame teletext onto an otherwise standard LASER videodisc and to use the teletext as a source of text, data and the controlling program. [The BBC had already published a consumer videodisc - British Garden Birds - which included a teletext magazine that could be read by any teletext TV.] Philips were approached because they produced the only videodisc players made in Europe.

There was some concern that the error rates to be expected from full-frame teletext would be high. This would mean that the 23K per frame that could be stored in this way would be substantially reduced by redundancy for error correction.

Before this question could be resolved Philips suggested putting a CD-ROM type of channel onto the disc in place of the sound channels (not on the sound channels). The data channel would provide over 300 Megabytes of data on a disc side ... such riches! This format later came to be called LVROM. This had the advantage of making more efficient use of the disc space, since the content of Domesday was to consist largely of still frames for which audio was not needed. Later videodisc players used this space for digital audio, which is how things stand today. (PAL videodiscs only had space for one set of audio channels, whereas NTSC discs had space for two. This was because NTSC discs revolved faster so had greater inherent bandwidth: 30 times a second compared to PAL's 25.)

The project was produced firstly for the BBC Micro Master computer (not the traditional BBC B), with an RML Nimbus version following quickly. The software was written by Logica using BCPL, an ancestor of C and which used a compiled intermediate code that ran on a virtual machine, which made it very portable. We did flirt briefly with the possibility of running the discs on an Amiga, since there was a BCPL for the Amiga, but that came to nothing. The Mac was out of the question because of the nine inch screen and PCs were only seen in businesses, not schools. (I did get into hot water however when a computer journalist misreported me as investigating the Amiga as a new BBC Micro rather than specifically for IV. Caused a laugh at Acorn HQ apparently.)

Contents of the Domesday Discs

There were two discs in the Domesday Project package: the Community Disc and the National Disc. The basic difference between them is that the Community disc is map-based and shows Britain as seen by the people who live there while the National disc is topic based and provides an overview of Britain.

Schools, and later community groups, were asked to generate three photographs and 20 40-column pages of text about blocks of the country 4 kilometres East/West and 3 kilometres North/South. There are over 24,000 such blocks in the UK. The contributions from schools were organised through the local education authorities, who were each asked to nominate someone to co-ordinate the contributions in their area.

The subject matter was left up to the contributors, as was the style of writing. It was important to find out what they thought was important about their area rather than what the project, as outsiders, thought. This did lead to some curious anomalies. For the block containing York Minster, one of the finest cathedrals in Britain, and an area with a great level of tourism and historical interest, the three photographs show a lady ironing clothes, a row of terraced houses and a street scene. (This was possibly a reflection of the local lack of 'ownership' of a tourist city ... showing us that York is for real people and not just for visitors.)

The text was checked for legal reasons but no attempt was made to correct idiosyncratic spelling or grammar. There is a wealth of information about life in Britain as seen by the British which it would be impossible to get from another source.

The National Disc is a more varied disc, utilising a different range of techniques for production and presentation. It contains sets of photographs, some professional and some from the national photographic competition, and there is a range of text from various published sources such as newspapers and magazines. There are nine surrogate walks, enabling the user to explore an environment from the keyboard of the computer. You can look and move around and look at details by clicking on things. One of these environments is generated by computer graphics and is an art gallery acting as an index to images on the disc. The pictures on the walls can be clicked on to lead to related images. This was our primitive version of non-immersive virtual reality (in 1986) and, as I produced the walks, I would have died to have QuickTime VR. One of the walks goes around a Barret show house, and our original plan was to populate this house with the best-selling items of consumer goods of all kinds, from groceries to beds to electronics. Clicking on a particular item would have taken you to a data set on the market for that particular kind of good. Unfortunately there wasn't time to undertake this, but we did photograph every single item in a Spar grocery store (in West Ealing) complete with its price; and this photo set is on the National Disc (but not part of Domesday Reloaded).

The National Disc also contains a set of seven short moving sequences of news and sports events from the UK in the years 1980 to 1986, starting with the enthronement of the then new Archbishop of Canterbury and finishing with the 13th Commonwealth Games in Edinburgh and including coverage of events ranging from the Falklands conflict and the miners' strike to the Wedding of Charles and Diana. (Looking back on this footage from the 21st century, one thing that strikes me is how much coverage there is of rioting in England and the 'troubles' in Northern Ireland.)

Finally there is a variety of statistical data about Britain. This includes the results of the 1981 census (the most recent at the time) as well as information ranging from causes of death to what the average Englishman does with his time on a Saturday afternoon.

These were the kinds of items which were available on the National Disc:

• Text with or without illustrations
• Photo sets
• Data to display as bar/line graphs
• Data to plot over a map
• Surrogate walks
• Film clips

Using Domesday

The system was operated using a trackball. Some people set up a touch screen, and that would work as well. The BBC Micro was not designed to have a point-and-click environment, but it could be programmed to use one. The system was designed with a menu bar ... at the bottom of the screen ... and a status bar at the top. The bar could be removed by clicking a button at its far left, leaving the small button behind. Clicking again returned the menu. Providing a mouse or tracker-ball interface for a BBC micro was relatively simple and only involved a small amount of machine-code programming to intercept the 'clicks' from the wheels in the device and use this to move an on-screen pointer. By 1984 there were several mice available for the BBC but we chose a tracker-ball mainly because it was felt more suitable for libraries and schools (it could be fixed to a table).

The Local Disc was based around maps, licensed from the UK Ordnance Survey. The disc contained the whole of the 1:50000 scale series of its day and many larger and smaller scale maps. Using these maps the user could navigate around the country, see where there are pages of text and photos, and look at them. The number of maps was such that the whole of the UK couldn't fit on one disc side, so the user would have to turn the disc over to move between the north and south of the UK. The boundary between the two sides, the so-called Domesday Wall ran from Lancashire to Yorkshire. [It gives some idea of how south-centered the 'official' view of the UK is, when the North is actually only halfway up.]

It was not straightforward to map the UK on the discs. The UK actually has maps based on three different national grids. Most of the country is mapped on The National Grid but Northern Ireland is on the Irish Grid and the Channel Islands were on the French Grid. [Jersey and Guernsey now have their own, separate grids.] For practical reasons these grids have slightly different definitions of North. This added to the complications of being able to operate a system where you could enter a grid reference and go quickly to the 1:50 000 map that included it. There was a gazetteer as well, so that you could enter a place by name and quickly see the map of it. On those maps you could measure distances and even areas. I sometimes used the system to work out mileage for my expenses :-)

The National Disc presented a more textual interface, but an extremely talented consultant named Martin Porter designed a natural language parsing system so that queries could be entered easily. [This made extensive use of what is now known as the Porter Stemming Algorithm.] With the huge amount of information on the discs, the number of hits was often large. As well as this query interface you could go through the hierarchy directly, following the tree and branch structure. There was the Landscape of Knowledge interface for browsing picture sets: this was the one built as an early version of non-immersive virtual reality. [The Bosch FGS 4000 computer graphics system used cost as much as a luxury car and, of course, is much less powerful than my mobile phone, let alone the computer I am using to type this web page. But it was ever thus!]

Much of the statistical data could be plotted over maps, using the translucent overlay facility provided in the hardware. You could choose the data set and the area to plot it over. It was even possible to correlate two data sets although this is something care is needed over since it is possible to compare two unrelated sets of data and find a spurious correlation. For this reason some comparisons were not allowed, otherwise you might find a correlation between car crime and a lack of red admiral butterflies. Data displayed as bar and pie charts could also be viewed over time and this gives rise to the famous dancing bar charts. Facilities like these are now commonplace but in 1986 this was something of a revelation.

The Structure of the Domesday Data

Domesday was conceived as a hierarchy with different facilities available at different levels. This produces a structure chart similar to a family tree, starting at the top, or entry, level. Alongside this the information contained within the National disc is organised within a hierarchical thesaurus. Every piece of data or picture set or surrogate walk or film item has a place within that hierarchy and was also given a set of keywords to enable it to be found during a search. The size of the Domesday database made it essential that every item was keyworded and had a place in the thesaurus; after all, the best database in the world is useless unless the items can be easily retrieved.

This is the tip of the hierarchy, of the iceberg as it were. It shows the main categories into which the information was structured and follows Culture in a little more detail.

                     Entry Point
        __________________|_______________
        |         |         |            |
     Society The Economy Culture The Environment
               _____________|___________________ ....
               |            |            |
            Tourism
     __________|_________________ .....
     |           |          |
                        Holidays
                ____________|____________________ ....
                |           |            |
         Holiday Patterns
_________________|_______________________________ ....
|                     |               |
Text:                 Data:           Data:
Adventure Holidays    Destinations    Expenditure on
on Safari             of Holiday      Camping/Caravans

Below are listed the top two levels of the hierarchy in their entirety. As you can imagine, trying to illustrate any lower levels in any detail would be impractical.

The Domesday Videodisc Player

After a two-year design and discussion process the laser videodisc player that Philips finally produced was the VP 415: The Domesday Player. In order to produce a plug-and-play system that any school could use the machine had special facilities. In this technical discussion you should remember that the videodisc was going to produce PAL video, and that the BBC Micro produced a PAL-like signal that could be shown on an ordinary TV set. Since the BBC Micro was the primary platform chosen for Domesday, this meant that both key hardware components of the system were producing video. You should also note that, at this time, the video standards on PCs were very similar to NTSC. A version of the project was made that ran on a Research Machines computer as well, since these were also common in schools.

Before computer graphics can be overlaid on a video signal the two must be synchronised. This process is called genlock and in the majority of interactive video systems the computer (or at least its video board) was genlocked to the video. Because the original plan was to enable potential customers to use their existing BBC Model B microcomputers, and there was a wish to avoid modifying micros in schools, it was decided to genlock the player to the computer.

Besides genlock, the player also carried a video-mixing board to combine the computer and disc pictures. This combined output was only available on the RGB output of the player since there was little point in degrading the combined image by coding it back into PAL, although of course the disc already contained PAL coded pictures. Under software control this output could be any of the following:

1. A hard key of computer output into disc video with any portion of the computer graphic that is not black replacing the corresponding part if the disc image.
2. As (i) but with a bleed of the disc image through the overlaid graphic. This produces a transparent overlay which was used for plotting data over a background map.
3. A highlight mode whereby the disc video is depressed in brightness for any portion of the picture where the computer output is black.

There are also modes giving clean output of either the disc video or the computer video.

In addition, because this keying was to be done in RGB inside the player it was possible to use an optimised PAL to RGB decoder and so improve the quality of the still frames. When a LASER videodisc sits on a still frame the pickup reads a video sequence consisting of only two fields. This is not enough to produce the full colour sequence needed for true NTSC (which needs four fields) or PAL (which needs eight fields). In conventional players this problem is overcome by processing the video in such a way as to rebuild a full colour sequence but at the expense of bandwidth and therefore picture sharpness and, usually, introducing artefacts. On a PAL disc the moving bandwidth is 5 MHz whereas this drops to nearer 3 MHz on a still frame together with much ringing on vertical edges. Since Domesday needed clear map images it was decided to include a special PAL decoder which could cope with a two-field colour sequence and give a good image.

The result is that from the RGB output (on the SCART socket) of the VP415 a sharp 5 MHz picture is produced even on a still frame. The player does have a PAL output (labelled CVBS) but this has the lower bandwidth and does not output anything but the videodisc video: no computer overlays at all. So, copying from the videodisc player gives nowhere near the potential quality of the images.

The specification of the data channel provides 6 kilobytes for every frame of video. Because of the interleaving technique used to error-protect the data it is not possible to sit on a still fame and read any of that 6 kilobytes; the system has to play across the frame to read both the data and the redundancy required for error-correction. The algorithm used in the filing system was clever enough to match data about a particular still video picture with that picture on the disc, so that the relevant data was actually stored in the portion of the data coincident with the picture which minimised disc accesses.

The data channel appears on a SCSI bus ... and SCSI had only just been standardised at this time. This means that the player looked to the host computer like a very large, somewhat slow, read-only hard disc. We had never used a CD-ROM at that point. A read-only version of the BBC Micro hierarchical disc filing system, now called VFS (for Videodisc Filing System), is used to access the data in a way that adhered to the filing system standards. In this way any code written for one filing system could be used with another, providing both use the same subset of the whole standard. Read-only systems obviously do not have sector write primitives or means of modifying file attributes for example.

Control of the player could be carried out down the SCSI, or alternatively using the RS232 bus on the player. However, once the system gives control over to the SCSI interface the RS232 was disabled so as to avoid conflicting commands. It was also possible to control the player from an infrared remote handset. The commands from this handset could be routed through the player to the host computer, allowing the handset to be used as a remote input device.

The SCSI interface was a long and narrow card which was added to the BBC Master and fitted in the place reserved (in theory) for a modem card. There was a ribbon cable connecting the card to an IDC connector on the back of the player. The SCSI card and VFS ROM were the only additions to a standard BBC Master Micro if it was the model with a coprocessor. The BBC was unusual in that you could buy coprocessor units or cards which provided a second 6502 processor with its own RAM. The two 'halves' of the computer communicated via a proprietary Acorn bus called 'The Tube'.

What Happened to Domesday?

In November 1986 the discs were published by BBC Enterprises. They were available as a package together with the necessary hardware for around £5,000, which made them a little expensive for many schools (it was the cost of a small family car) and over twice what the project believed they were to cost when we started. By this time the British Government had changed its policy from subsidising hardware in schools to subsidising software. The best efforts of Philips and Acorn and even the help of ESPRIT funding from the European Commission could pare the cost down no more. After a while, a company called Simnett's found a way to offer Domesday Systems at a lower price and more were sold. The targets were schools, universities and libraries. There is one at the Ontario Science Centre (if it is still in use you will need to ask to use it as it was never intended as a permanent exhibit), one was donated to the Smithsonian in Washington, and The Science Museum in London displayed one showing the Community/Local disc on display labelled as a Geographical Information System. If you know where one is available for either public or research access then please let me know because I still occasionally get asked.

Other communities looked into producing Domesdays of their own. Statistics Canada planned a project called Jean Talon but it did not come to fruition for lack of funds. This was a bold plan to get 20th Century Canada to leave a message for the new millennium. There was an idea to do one for the State of Maine but, again, I think it proved too expensive. And I occasionally got messages from people asking if anyone was re-doing or republishing Domesday for the Millennium. Ironically, the BBC struck good deals for use of material on the discs which would no longer apply, and contacting thousands of ex-schoolchildren to get permissions might be arduous. It would be interesting to contemplate a World Wide Web approach to the snapshot of the nations ... a kind of super-blog.

If I personally have one regret about Domesday it is that many of the people who contributed to it never got to see the results of their efforts. For that reason alone I am delighted to see it becoming more widely available now that the real world has caught up with our technological vision. It would also give us the chance to put in a Quit button. That was one thing we forgot. [Actually this was a myth – as I have been reminded since – a quit button would have been troublesome in schools and libraries ... so the omission was deliberate after all.]

The View from the next millennium

A 2002 article in the UK press [Observer March 3 2002] awakened interest in the fate of the BBC Domesday Project, majoring on how the original Domesday book was still readable after (at the time) 925 years while our 15 year old one was not ... unless you have the original computer/videodisc system and it still worked of course. I can't avoid asking how many of the journalists making this comment have actually tried to read the original book, with its hand-written Latin, but the underlying point is valid.

It would be nice to be able to look back on the Domesday Project and say "If we'd done it this way then it would still be around". This is especially interesting because a number of people have commented on the special hardware that was needed to play the videodiscs. What might we have done differently?

Not used the BBC Micro

Was there an obvious alternative? The intended market was schools and they had either BBC Micros, Research Machines or Sinclair computers. As it happens RM are still going strong and have a good presence in UK schools and even got a two-letter domain name in rm.com. Acorn, however, were partners with the BBC and were a featured part of the BBC's Computer Literacy Project in the early 1980s. At the time they also had arguably the most innovative machine.

Used a standard videodisc player

Originally I investigated using an unmodified videodisc player and storing data as full-frame teletext. The BBC's Telesoftware Project was regularly transmitting software using teletext on the BBC's Ceefax teletext service. (Teletext, for those non-Europeans in the readership, is a data signal transmitted in the vertical blanking of a TV signal and used to produce small pages of text and basic graphics. The BBC's Ceefax service started in 1972 and regularly attracted around 15 million users a week by the time it was closed with the advent of digital TV. Pretty-well all UK analogue TV channels broadcast a teletext service of one kind or another ... and so did most European ones.) There was a teletext adapter for the BBC Micro and I had produced a demonstration application using a BBC videodisc called "British Garden Birds" which included a teletext magazine. This application pulled digital data from the videodisc using the teletext adapter. Unfortunately it was a little temperamental although I did successfully demonstrate it to the BBC Board of Governors in 1984. The Domesday team did consider this technique but we were concerned about the difficulties of error correction - a teletext transmission refreshes every minute or so and can get around occasional corruption this way - and Philips suggested using a more elegant base-band technique. As it turned out, the BBC Micro Teletext adapter and the Telesoftware service were dead even before the Domesday Project finished!

Use CD-ROM

The then head of BBC Records, Humphrey Walwyn, told me about CD-ROM in 1984 but by then we were well down the path to LV-ROM. There was also no standardised way of handling the data on a CD-ROM. The killer was that we wanted to include thousands of still images and, in those pre-JPEG days, the best we could have done was an 8-bit adaptive palette at TV-resolution still taking up 300 kilobytes per image: only 2,200 could have been stored on a CD-ROM. (It's worth noting that VGA as a display standard wasn't introduced, by IBM, until 1987 and was based on the resolution of the American NTSC television standard.) Digital video was totally out of the question. None of the numbers added up. And to cap it all, using videodisc gave us full-colour rather than just 8-bit.

The BBC Domesday Project was way ahead of its time. If the Internet had been around then the data could have been collected that way and the results made available that way; although I suspect many of the continuing copyright questions over the content would still have arisen.

Preservation & Emulation - Domesday Redux

This web page has received a regular number of hits over the years and I have received many emails about the project. The most common questions are whether the material on the discs will be republished and how can people see it? If you read the top of this web page then you'll have an answer already of course.

The first visible manifestation of a reappearance of the BBC Domesday Project was achieved in a project called CAMiLEON, which was a research project that investigated emulation as a digital preservation strategy and was based at the Universities of Michigan and Leeds. [CAMiLEON web site ... with supreme irony this is now only available via the internet archive]

CAMiLEON demonstrated their work at the Leeds and, although not complete, they were able to run much of the three interactive videodisc sides on a contemporary PC, including the translucent overlays used to map statistical data geographically on the National Disc (aka Side 3). This was done by emulating the Domesday Project hardware within Windows so that the original programs will run. I should stress that you can not go to the CAMiLEON Project to view Domesday. They are not able to put it on public display and the project team have moved on. (What does that remind you of?!)

Two freelance programmers also worked (independently) on reverse engineering the data file stored on the videodiscs in order to enable the data to be displayed on a 'modern' Windows-based system. Neither of these attempted to recreate the original user environment and in one case deliberately looked afresh at how Domesday could run without being limited to a TV screen resolution. That resulted in the new version 'preserved' at the UK National Archives and (for a while) through domesday1986.com but that version is now offline. The domain has been taken over by the Domesday Special Interest Group, which includes a useful blog on Domesday and related matters.

Alongside all this work there was a project to preserve and archive the video content. This work, which I managed for the UK Public Record Office (as was) in conjunction with the BBC, produced archival video masters of the videodisc contents that have, as far as we can predict, a reasonable shelf-life using broadcast video technology.

The original PAL analogue video (from master tapes used to produce the original videodiscs) was converted into 'clean' digital component video using a specially designed PAL decoder made by BBC R&D. This so-called Transform Decoder (invented by Jim Easterbrook with hardware development by Richard Russell) has also been used to make DVD masters of old Dr Who episodes, so we're in good company. The 'new' digital masters will be used to 'preserve' the video material as well as providing a good reference source for stills and video which can be used to view Domesday in the near future.

In mid January 2003, the original 1-inch C format analogue videotapes - to be exact, masters for sides 1,2 and 3 and a safety-master for side 4 - were copied onto D3. D3 is an uncompressed PAL (composite) broadcast videotape format and is, for the project, an intermediate format enabling the PAL video signal to be moved from somewhere where it can be carefully played from the original, and potentially fragile, 1-inch tapes to somewhere where the PAL can be carefully decoded into component form for preservation. Ironically D3 is an obsolete format in a component world.

The master tapes for the Community Disc and for one side of the National Disc had been stored safely away since 1986 by, firstly, the BBC's Open University Production Centre and then by Domesday head Peter Armstrong. (Neither in his loft or his garage but in his office at home.) A safety master of the fourth disc side was also found in the BBC Worldwide vault. None of these tapes had been played for almost 17 years until the BBC engineer in Brentford, at the BBC Archives, laced them onto an Ampex C Format machine for the first stage of the copying.

Three of the four tapes were perfect, but one tape showed some slight signs of deterioration, akin to fine scratching. (This was very noticeable in the RF signal actually recorded on the tape, which was displayed on a scope on the videotape machine as the tape played.) It seemed as if the tape improved each time it was played and we could see that it was not noticeably shedding its oxide and clogging the heads. So we took the decision to play it through completely once before copying it. A few extracts which seemed to be the most affected were copied and closely checked for visible artefacts. Apart from a little banding on some areas of strong colour in the titles and help sequences, which were duplicated elsewhere, we could see no problem and so the tape was copied. A couple of extra sequences used on the discs (including the computer generated gallery and an extended version of the 35mm title film shot in Chester) were also copied onto D3 from their first generation tapes, as they were also found.

At the end of January 2003, all these D3 videotapes were copied via the Transform Decoder onto Digital Betacam (in duplicate) for archiving and onto DV for viewing. The entire process was done in the digital domain. Although Digital Betacam is a (lightly) compressed format, there is no other practical way of storing such a quantity of video while being confident that in ten or twenty years time it can be replayed. It is also fair to say that the compression is a maximum of 2:1 since it uses DCT-based compression (as do JPEG and MPEG) and in the case of the relatively 'soft' images in television there are few high order DCT components and so less compression - if any - is needed to fit the recordings into the DigiBeta data rate. The end results showed no compression artefacts.

In some places there were artefacts resulting from dropouts on the original 1-inch analogue tapes. These show as repeats of lines or parts of lines as the Ampex dropout compensator - designed of course for moving video - did its stuff. Sometimes scratches on portions of the tape show as short light-coloured horizontal lines, one line/pixel deep. The videotapes show none of the patterning and speckling that is clearly visible on the videodiscs. [Some argue that this means that the archived video doesn't really preserve the original look of the video. I can see that flaws are a valid part of any artefact but this must not be used as an argument for losing higher-quality sources.]

Although this whole chain of events involves changing the video format from PAL composite to digital component, composite video is not seen as having a long-term future and the expertise to best convert PAL into component video will not last for ever. The Transform Decoder was specially reprogrammed to cope with the single frames on the Domesday videotapes since this involved a slightly different technique from that used with moving pictures. (This is related to the eight-field sequence in PAL.) The colour space, what is generally referred to as YUV or YPrPb, remained the same.

The size of the digital Domesday images are 702 by 576 pixels, conforming to the digital broadcast standard for 625 line TV. Of course the scanners and cameras used to grab the maps and slides were analogue devices and produced a truly analogue signal - this was in the days before pixel-based CCD scanners and cameras - so the concept of pixels is not straightforward. The colour subcarrier embedded in the PAL video signal has a fixed frequency (4.43 MHz) which imposes a pixel-like pattern on the video, although it is shifting in phase all the time as it shows the colour values along the TV line. The digital sampling frequency is 13.5 MHz, which is more than twice the subcarrier frequency (ref Mr Nyquist). The aspect ratio of the image is, of course, 4:3, but TV pixels are not square, whereas computer pixels are square.

This is not a typo: it is 702 and not 720 (just do the maths: 13.5 megahertz along a TV line of 52 µs duration). Even though digital consumer camcorders typically produce images that are 720 pixels across, a correctly set-up 625 line broadcast digital image should only be 702 pixels across. The specification does not guarantee that the remaining 18 pixels will be displayed or need to be displayed and they are usually left black. Video digitised from a PAL source should only be 702 pixels across to preserve the aspect ratio. To display digital video on a computer screen it has to be stretched horizontally ... but by different amounts depending on the source. The result of this is that the DV viewing copies of the Domesday master tapes have narrow black bands down either side. An alternative is to keep the 702 pixels across and shrink the image vertically. This has the added advantage of slightly reducing noise and is what I in fact did in producing 'full size' images for display. The advice from the BBC was that horizontal sharpness was perceptually more important than vertical sharpness so I decided not to resize the images horizontally.

DigiBeta was a pragmatic choice of archiving format, and I am under no illusions that it will last forever. But I am confident that it was the right medium-term choice, partly because we are using a well-defined standard format (ITU Rec 601) and we remain in the component colour space rather than converting into RGB (but this is a relatively minor consideration). Also, broadcast engineering has always taken a longer-term view of technology than computing. Even if we have to go to a special facility to use a DigiBeta machine in 2020, it is a good bet that there will be one to use. It is fair to say that 2-inch monochrome quadruplex videotapes, which could be almost 50 years old, can still be replayed somewhere today, which would give 'us' until 2050 to transfer the digital information onto a newer format. I do not have any such confidence about most other forms of storage. Servers are not an obvious alternative since these are still backed up and archived onto streamer tapes (although I have heard strong arguments that such streamer tapes are a better choice than videotapes, and even hard discs may provide a useful medium-term archiving medium). Do you still have 8mm Exabyte tapes on your shelves? DVD-ROM might be an answer, especially since DVD discs lend themselves well to 'diversity preservation' being easy to duplicate and small enough to store easily. It is also such a widespread medium that we should expect it to have some longevity. You will note that I say DVD-ROM and not DVD, since compressing the images would degrade them further.

Storing a complete uncompressed set of Domesday images was initially considered too big a task. Now, with the cost of storage having dropped dramatically, the entire stills content of the Domesday discs stored as 24-bit PNG files comes to around 200 gigabytes, which can now easily be stored on cheap hard or even solid state drives. The hour or so of movie sequences (help, titles and side 4) can be stored the same way: this may well happen at some time in the future. We at least have some leeway in doing this.

[While DigiBeta provided a useful medium-term method for archiving on tape, the situation is different when archiving high quality video onto a digital medium. I have been told by engineers at BBC Research that by using what is called 'long GOP MPEG' you can store moving images at the same perceived quality as DigiBeta but in much less space (possibly a tenth). 'Long GOP' refers to the number of frames contained in a 'group of pictures' in MPEG. Each GOP has an I-frame at each end—an I-frame is self-contained and needs no other frames to be decoded—while the frames in between the I-frames are 'built' by referring to the I-frames and to information on the changes (looking both ahead and behind) between those I frames and the current frame. This approach would not work for the Domesday video because it consists mostly of thousands of individual stills. These would have to be stored as MPEG I frames, were MPEG to be used. Some archivists are understandably wary of long-GOP because of the danger of corruption to data between the I-frames. Obviously, the longer the GOP the more video is potentially lost when the data in a GOP is corrupted and lost. You could counter this concern by saying that it is easier to protect data stored digitally using parity, RAID or other diversity-based systems, for example, and that this is not too difficult to achieve. A third point of view is that storage is becoming ever-cheaper and that attempting to archive by using anything other than uncompressed data is a false economy. In this case data tape storage is an option for storage but even uncompressed data can 'rot' either due to deterioration of the medium or even the action of cosmic rays! I should add that diversity is still required since I would bet that more records have been lost due to fire or water damage than due to 'rot'. Fêtes vos jeux mesdames et messieurs!]

One set of the DigiBeta tapes was sent to the BBC and the other is held at the UK National Archives/Public Record Office. DVD-ROM sets of pictures were sent to the BBC, PRO and to Long Life Data, for incorporation in the PC version. A set of the images from the three interactive sides has been transferred into individual uncompressed PNG files from one of the Digi-Beta clones with a view to higher quality material being archived.

In considering why this project—big as it was—became 'lost', a key issue is 'ownership' ... in a psychological sense more than a legal one (although the legal one can't be ignored). Since the BBC made its interactive production team—the extension of the Domesday Project team—redundant in 1990 there was no continuity inside the Corporation to make sure that the project was preserved, or even to remember it. It wasn't a TV or radio programme and so didn't fit into the usual pigeon-holes. The half-inch ANSI data tapes from which the Domesday videodiscs were made, built using VAX computers, are almost certainly lost for real. Fortunately each copy of the videodiscs contains a set of the data which, being digital, is in 'pristine condition' on any disc that can be correctly read. The same is not true of the analogue video content. If the master tapes had not been found and the technology and expertise to replay them had not been hanging around by it's fingertips then, reluctantly, the video on discs would have been the only source.

This is not the whole story, of course. While the archiving and preservation of the Domesday video content is useful we have to remember that the video is only useful in the context of the whole project. For a complete preservation, the video has to be incorporated into a preserved (and sustainable) Domesday Project Whole, no matter how long the DigiBeta copies last. As Paul Wheatley from CAMiLEON rightly says ... "We must not make the mistake of thinking that recording on a long-lived medium gives us meaningful preservation".

If you're interested further, an edition of the BBC World television programme 'Click OnLine', which was broadcast in February 2003, had a feature on digital preservation and Domesday in particular ... and I'm interviewed in it! Unfortunately the current on line Click archives only go back to 2004. Domesday Reloaded has produced some more examples for you. See the top of this page.

It seems there is a big international plan underway to try and work out just how we can avoid losing access to digital archives. The pace of technological change is now so fast that we can't necessarily expect our current data storage media to be usable in 15 years. The UK Public Record Office hosted a conference in April 2003 called Practical Experiences in Digital Preservation, at which the Domesday Project (both original and 'resurrected' versions) was shown. Audio of this event used to be available on the National Archives web site but has since been taken down.

I could argue that the only way to avoid this is for data to be regularly copied onto whatever is the current storage medium of choice. This isn't only a digital problem. Vast archives of valuable television programmes stored on obsolete videotape formats (2-inch quad, 2-inch helical, 1-inch helical in 3 different flavours, U-matic ... to name but a few) have been and are being copied onto current tape formats and even onto servers 'as we speak'. Legacy programming stored in composite (NTSC,PAL and SECAM) formats almost certainly needs to be converted to component or RGB formats and this has to be done without introducing new artefacts and ideally while removing some of the composite ones, since they usually follow a recognised pattern (often literally). It's not only Domesday's videodiscs that are obsolete: the videotape machines the masters were recorded on have also gone the way of the Betamax ... and the Edison cylinder player.

It would be nice if we could avoid all this 'mechanics' altogether. If it were possible to produce a suitably high-resolution bit-map 'image' of the magnetism on an analogue recording tape - an image 'ribbon' - then it ought to be possible to analyse this image using software to 'play back' the recording that was on the tape. This would pass most of the problem of handling different tape formats from the physical domain into the software one. But how could we produce the image in the first place? Ideally a simple linear array of heads, over which the tape could be passed at any practical speed, would do the job. This should be easily within the capability of present and future engineering: unlike the intricacies of rotating heads on drums. Unfortunately a linear array might not be able to read the tape (at least that's what tape experts have told me) so some kind of scanning system would be needed still. But if you could produce a simple scanning system that could read any tape - from a few millimetres wide to two or even three inches - then a truly universal tape playback machine would be possible. I like the linear idea best: it would be kinder to often-fragile tapes.

The motion picture industry can be glad it has 35mm film (but for how long?). Apart from getting rid of the explosive nitrate film the means of storage and distribution of movies has had this common thread for over a century. But what do we do with Digital Cinema? Moving from celluloid to digital will mean cleaner images and better distribution ... but the Lumière brothers' movies can still be played after a hundred years. Can we guarantee that a digital stream of, say, 'Toy Story 3', will be accessible in 2100? I am told that the best method for storing a feature motion picture is to produce a set of three 35mm celluloid films, one for each primary colour, and keep them in cold storage. This is exactly the system used for Technicolor masters for decades. Finally, it is sadly not safe to assume that the mass production of DVDs will preserve movies any more that the mass production of gramophone records has preserved their contents. That presumption favours the popular. On top of this, there are the implications of copy-protection. This leaves us with a scenario equivalent to the Mona Lisa trapped forever behind an opaque and impenetrable brick wall.

For an alternative perspective please check out an article by Richard Wright, from the BBC Archives, on an EC Project called PRESTO. This project looked at the archive and access issue from the broadcasters' perspective but has many useful analyses for a more general audience.

Perhaps data storage is too important to be left to 'everyman'. Rather like the role of the librarian in the book 'The Name of the Rose' it is not for us ordinary monks to know where the information is actually kept: it is enough that we can be given access to it when we need it. Have I just argued for cloud storage?

Let's not decry Domesday as a waste of time and money. It is a salutary lesson in technology creep and we need to know how to cope with it in future. Watch this space!

The Geograph British Isles Project

Although I have absolutely nothing to do with it, I must recommend the Geograph British Isles Project, initiated by Gary Rogers. This is an online community that is photographing every one-kilometre square of the UK, based on Ordnance Survey mapping (and sponsored by OS). (It's interesting to see that the site is being archived by the British Library as part of their UK Web Archive project.)

At the time of writing they say that '13,317 contributors have submitted 6,711,758 images covering 280,772 grid squares, or 84.5% of the total'. Quite an achievement ... and a worthy successor to the BBC Domesday Project. Now if only they had text for each of those squares ...