Adding references

What a RIS dataset looks like

The main advantage of RIS are:

• The tag set is well suited for managing references and offprints used by scientists. There is no overhead for library capabilities.

• As a plain-text, tagged format it is easy to create and edit with basically any text editor.

To get started, we'll assume you have some reference data in RIS format handy. This could be your existing Reference Manager or EndNote database exported to RIS. However, for your first experiments the example file shipped with refdb is just fine. This file is usually installed as /usr/local/share/refdb/examples/testrefs.ris (ask your administrator if there is no such file).

At first you should have a look at the data, just to know what they look like. Use either a pager or a text editor to display the example file:

~$  less /usr/local/share/refdb/examples/testrefs.ris
            
            TY  - BOOK                                    
            ID  - smith1975metalloporphyrins              
            T1  - Porphyrins and metalloporphyrins        
            A1  - Smith,K.M.                              
            Y1  - 1975///                                   
            KW  - Porphyrins                              
            KW  - Metalloporphyrins
            KW  - Spectrophotometry [methods]
            KW  - spectroscopy
            RP  - NOT IN FILE                             
            CY  - Amsterdam                               
            PB  - Elsevier Scientific Publishing Company  
            ER  -                                         (11) 
         

Ok, I've cheated a little. This reference is not the first and only one you'll see if you run that command, but it is the shortest one, good enough to discuss the general anatomy of a RIS dataset (in order to see this reference in the real example file, scroll just about halfway down).

We can easily discover these main features of the RIS format:

• A RIS file can contain one or more datasets.

• Each dataset starts with an empty line (more precisely, with a linefeed character). This also means that each file containing RIS datasets starts with an empty line.

• The tags start at the beginning of a line and consist of two uppercase characters, followed by a double space, a dash, and another space.

• The TY tag is always the first one, and the ER tag is always the last one in each dataset

• The sequence of the other tags is arbitrary with one exception: The sequence of the author fields (A1 or AU) determines the sequence of the authors in bibliographies or citations. The sequence should follow the sequence in the original publication.

Now let us look at the RIS dataset shown above. It contains the following tags:

The following list gives an overview over all available tags and their use.

Creating a RIS dataset from scratch

As noted previously, you can use any text editor that creates Unix-style line endings (linefeed) to create and edit RIS files. refdb ships a ris-mode for Emacs which makes the task a little more pleasant. Ask your administrator whether this mode is available on your system.

Creating a dataset is basically monkey business. The only intelligence required is to use the correct tags for your chunks of bibliographic information. Use the example RIS file as guidelines for the most common reference types journal article, book chapter, and book. We'll look at a few issues related to these references first.

The first issue is certainly the type of the reference. There are pretty clear cases if you look at the list, but there's some bordercases too which are not covered by the predefined types. You should keep in mind that refdb does not restrict the fields available for a particular reference type. You can fill any available field for any reference type. The only restriction is that the bibliography styles for most reference types use only a subset of the available fields. E.g. a bibliography entry of a journal article will not show a series editor even if you filled in the A3 field, whereas the bibliography entry of a book chapter might show the series editor if the book the chapter is published in is a part of a series.

The general rule is to use the closest matching type, to be consistent in this decision (all similar bordercases should use the same type), and to use the GEN type if nothing else helps. Most bibliography styles display all available fields for the GEN type.

To people new to bibliographic software, the various levels of titles and authors is often confusing. RIS offers three levels of authors:

Lets consider a few examples. If your reference contains a journal article, published in some scientific journal, the AU fields contain the names of those who wrote the article. The same holds true for the authors of a chapter published in a book like "Methods in Enzymology". The chapter authors would be in the AU fields, the volume editors in the ED field. The editors of the whole "Methods in Enzymology" series of books would be entered in A3 fields. However, if your reference points to one particular volume of "Methods in Enzymology" as a whole, you'd rather put the volume editors in the AU fields. The same logic holds true for the title fields:

Using our previous example, an article published in "Methods in Enzymology" might have a TI field "An apparatus to turn urine into gold". The T2 field would be the title of the volume, e.g. "Alchemy and related techniques", whereas the T3 field would contain "Methods in Enzymology". However, if your reference points to the "Alchemy and related techniques" volume as a whole, this title would go into the T1 field.

Retrieving datasets from PubMed

The primary source of reference data in the biomedical field is the PubMed database maintained by the National Center for Biotechnology Information. This section shows the simplest and most common way to retrieve bibliographic information about interesting articles from this database using a web browser (other methods use web service clients or email subscription services, but this is beyond the scope of this tutorial).

After visiting the site with your favourite web browser, select "PubMed" from the drop-down box called "Search" and type a query in the provided field. Something like "Doe J 2002" to find articles published by J. Doe in 2002. After hitting "Enter" you'll receive a list of publications matching your query. Select the ones you're interested in by clicking the check box right next to the publication (convenience beating logic, you can also check none of the boxes in order to retrieve all publications). Select "XML" from the drop-down box next to the Display button and hit the latter. You'll receive the list of the publications in the Pubmed XML format. Now click the Save button on that page and save the information to a plain-text file, e.g. pm001.xml. You could then return to the search, run a few more queries, and save your results in additional files according to this pattern.

We'll use the Perl script med2ris.pl to turn our XML data into RIS data. This tool either reads Pubmed data from standard input or from files specified as arguments. The result will be sent to standard output, so you can either view it with a pager or write it to a file.

~$ med2ris.pl < pm001.xml | less

This command converts the data in the file pm001.xml and displays the result in a pager.

~$ med2ris.pl pm*.xml > pm.ris

This command converts all files that match the pattern pm*.xml, like pm001.xml or pmnew.xml, and writes the resulting RIS datasets to pm.ris.

Now you should add your personal information to each dataset, as outlined above. Then you could go ahead and add the references to your default database with refdbc:

refdbc: addref pm.ris

Importing BibTeX datasets

If you have a BibTeX database that you want to import into refdb, you'll have to convert these data to RIS first. This is again best done by using one of the converters shipped with refdb. The tool bib2ris will by default convert all standard BibTeX fields to the RIS equivalents. If you used non-standard fields in your BibTeX database, bib2ris can be configured to import these too, but this requires additional entries in the ~/.bib2risrc configuration file. The manual has all the details, but for the purposes of this tutorial we'll assume that you only use an "ABSTRACT" field as the only additional non-standard field.

Just like refdbc, bib2ris reads configuration files at startup which you can modify to permanently set some defaults. The syntax of the configuration file is the same as outlined above, but the only line we have to enter at this time is the following:

nsf_abstract N2
         

This will tell bib2ris to import your non-standard abstract field (this is case-insensitive, so your BibTeX file might use ABSTRACT or Abstract as well) into the N2 RIS field. Use the following command to see the results:

~$ bib2ris < myrefs.bib | less

This command will convert the contents of myrefs.bib in the current directory and display the result in a pager.

~$ bib2ris *.bib > myrefs.ris

This command reads the data from all *.bib files in the current directory and redirects the output into the file myrefs.ris.

Now there is a little issue with the data generated by bib2ris: they still contain all TeX markup that you may have used in your input data. If you want to use refdb only to maintain references for LaTeX files, this is probably ok, but if you want to use the data for SGML and XML documents too, it is necessary to strip the TeX markup before adding the references to the database. To this end, run the RIS file through the Perl script tex2mail shipped with refdb.

~$ tex2mail -noindent -ragged -linelength 65535 -ris < myrefs.ris > myrefs-notex.ris

As described in the previous section, you should now add your personal information and then use refdbc to add the datasets to your database.

Retrieving datasets from a Z39.50 server

Many libraries allow remote access to their electronic catalogs via the Z39.50 protocol. With a suitable client you can search the catalogs and retrieve the bibliographic information of interesting references to your local computer. For this tutorial we'll use the free client provided in the YAZ toolkit, although you could use any other client as well. One of the largest libraries accessible via the Z39.50 protocol is the Library of Congress. We'll try to find computer books written by some Mr. Knuth in this library.

The following command connects to the library using the host name "z3950.loc.gov", the port 7090, and the database name "Voyager". All this information is usually provided either online or in a printed pamphlet about electronic catalog access published by libraries offering Z39.50 services:

           ~$ yaz-client z3950.loc.gov:7090/Voyager
Sent initrequest.
Connection accepted by target.
ID     : 34
Name   : Voyager LMS - Z39.50 Server
Version: 1.13
Options: search present
Elapsed: 5.176489
Z>

Now you can go ahead and type a query. The full query syntax of Z39.50 is beyond the scope of this tutorial, but the following query retrieves entries with the authorname "knuth" and the topic "computer":

Z> f @and @attr 1=1003 knuth @attr 1=4 @attr 5=1 computer
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 28
records returned: 0
Elapsed: 5.187307
Z>

We've found 28 entries that match our search pattern. We could just go ahead and display some or all of them, but we'd like to write them to a file, so we let our client dump all retrieved references to the file knuth.loc.usmarc and make sure the data are retrieved as "usmarc" (again, the library should be able to inform you which formats are available). Other formats acceptable for refdb are "ukmarc" and "unimarc".

Z> set_marcdump knuth.loc.usmarc
Z> format usmarc
Z>

Now we retrieve all of the matching entries. The show command uses an argument like X+Y, where X is the record number where the retrieval should start and Y is the number of consecutive records to be retrieved. The data will be displayed on the screen and written to our file in the background.

Z> show 1+28
Sent presentRequest (1+28).
Records: 28
[VOYAGER]Record type: USmarc
[...real data left out for brevity...]
nextResultSetPosition = 29
Elapsed: 8.438264
Z>

Finally we can leave the client by typing:

Z> quit
~$ 
       

If you attempt to open the resulting MARC file with a text editor or display it with a pager, you'll notice a couple of strange characters. MARC is a binary data format which is not supposed to be readable as plain text. If you want to display the file in a human-readable form, use the tool marcdump (this is part of the MARC::Record perl module which is required for the marc2ris.pl converter shipped with refdb; if there is no marcdump on your system, ask your administrator):

         ~$ marcdump knuth.loc.usmarc | less

The structure of a MARC record is quite complex. It is divided into a leader, fields with three-digit names, indicators, and subfields. No need to understand it at this point, though.

refdb ships the Perl script marc2ris.pl which attempts to convert MARC datasets to the RIS format like this:

~$ marc2ris.pl knuth.loc.usmarc > knuth.loc.ris

This will convert the references we downloaded to corresponding references in RIS format which will be written to the file knuth.loc.ris. If you retrieved the data in a different format, use the -t command line option to specify the input file format: "marc21", which is equivalent to USMARC, "unimarc", or "ukmarc". Now you can proceed as described above and add the contents of this RIS file to your database.

What a risx dataset looks like

The main advantages of risx are:

• The XML format is a good target for transformation of other bibliographic data in SMGL or XML formats.

• XML can be edited using either a general-purpose editor or, even more conveniently, with any XML editor.

• XML datasets can be validated, i.e. checked for completeness and for an appropriate structure.

Now lets have a look what the same dataset we had in RIS format above would look like in risx:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE ris PUBLIC "-//Markus Hoenicka//DTD Ris V1.0.2//EN" "http://refdb.sourceforge.net/dtd/risx.dtd">
<ris>
  <entry type="BOOK" citekey="smith1975metalloporphyrins">
    <publication>
      <title>Porphyrins and metalloporphyrins</title>
      <author>
        <lastname>Smith</lastname>
        <firstname>K</firstname>
        <middlename>M</middlename>
      </author>
      <pubinfo>
        <pubdate>
          <date>
            <year>1975</year>
          </date>
        </pubdate>
        <city>Amsterdam</city>
        <publisher>Elsevier Scientific Publishing Company</publisher>
      </pubinfo>
    </publication>
    <libinfo user="markus">
      <reprint status="NOTINFILE" />
    </libinfo>
    <contents>
      <keyword>Porphyrins</keyword>
      <keyword>Metalloporphyrins</keyword>
      <keyword>Spectrophotometry [methods]</keyword>
      <keyword>spectroscopy</keyword>
    </contents>
  </entry>
</ris>

This is a complete file with just one reference (you could add more entry elements for additional references). As you can see, the entry is a lot more verbose compared to RIS due to the spelled-out start and end tags. However, modern XML editors compensate this verbosity with nifty feature like tab completion and automatic insertion of end tags.

You will notice three large blocks of data in this dataset:

• The publication element contains the bulk of the bibliographic data proper, like the title, the author(s), and the publication date. Simple entry types like the book we see here make do with one level of bibliographic information. Complex types need more than one level. A journal article needs the part element for the article proper and the publication element for the information related to the journal. A book published as a part of the series needs the set element for the series information in addition to the publication element.

• The libinfo element contains the information specific to one refdb user, like availability information and personal notes. As a matter of fact, a risx dataset can contain an unlimited number of libinfo elements, one per user of the system. See also the section about global and personal fields.

• The contents element holds the abstract (not shown here) and keywords.

For further information please visit the documentation of the risx DTD.

Writing risx datasets from scratch

Using your favourite XML editor, writing a risx dataset from scratch should not be exceedingly difficult. The editor should prompt you for required elements and attributes, and refuse to enter an invalid structure. See the example datasets shipped with refdb to get an idea what different entry types should look like.

Transforming SGML and XML bibliographic data to risx

This topic is somewhat beyond the scope of this introductory tutorial, but if you're familiar with SGML or XML transformations in general, this should not be too hard either. Each input bibliographic format will require a custom DSSSL (for SGML data) or XSLT (for XML data) stylesheet that transforms the data to risx.

Validating risx datasets

If you write risx datasets from scratch or develop your own stylesheets for SGML/XML transformations, it is strongly recommended to validate the results of your laborious efforts. refdbd uses a non-validating parser to map the risx data to the appropriate database columns. If your input data are invalid, the results might not be to your liking. Two tools come in handy to validate your input data: