Thursday, 27 December 2018

Database Programs - the Generic and the Specific

So I have been working on a Fighting Fantasy project. Before I or anyone can play the game I need to input the data for both paragraphs and monsters. Currently the SQLite database (fight.db) has 4 tables, each with a number of columns.
monster table
  • name (text, primary key)
  • skill (integer)
  • stamina (integer)
  • treasure (integer)

equip table
  • monname (text, primary key)
  • equipname (text, primary key)
  • equiptype (text)
  • lootable (integer)
  • adjust (integer)

paragraph table
  • module (text, primary key)
  • pararef (text, primary key)
  • paratext (text)
  • monster (text)
  • looseitem (text)

options table
  • module (text, primary key)
  • startpararef (text, primary key)
  • endpararef (text, primary key)
  • optiontext (text)
  • conditions (text)

I won't go into a detailed explanation of each column, suffice to say there are a mix of text and integers, and some fields are primary keys.
For those who are not familiar with primary keys, these are columns that are used to single out and identify each record. If a table has a single column as its primary key, each record in the table must have a unique value for that column. So for example, in the monster table there is one column as the primary key, name. Each entry must have a unique name value. If a table has 2 or more columns as primary keys, then the combination of those columns must be unique. For example, an entry in the paragraph table must have a unique combination of module column and pararef column.

To add new records, delete unwanted records and edit existing records in all four tables I have created two different programs using two different approaches. The first one took quite an abstract approach, treating the tables as unrelated and not really taking into account the individual needs of each table or each column. The first program uses the PRAGMA command in SQLite to retrieve information about each table, namely which columns are primary keys and also which are text or numbers. This means that the program won't try to input text values into columns for integers. It should also catch duplicate primary keys. The good news for this approach is that it is very adaptable, and if I change the tables or even use it on a different database it should still work. The program itself is 263 lines of code & comments

The second one is much more customised and takes into account the relationships between tables.
Although in Python a variable can represent a list, tuple or dictionary (a series of different values, called an iterable), in SQLite they don't use lists or other iterable datatypes - you create more tables and elements that would be in the list become records in the new table.
So if a monster has several items of equipment associated with it, rather than having a column called equip, there is a separate table. One column (here monname) says what monster record each item in the table is associated with. Here the value in monname should match up with a value in the name column of the monster table. Another column (here equipname) identifies each item associated with that monster. It is no coincidence that monname and equipname are the two columns that form the primary key for this table - the combination of monname and equipname should be unique and identify each entry in the equip table.
This is a similar situation to paragraphs and options - each paragraph typically has several options associated with it, though each option has its own characteristics associated with it.

So when entering a new paragraph, the program will ask whether the user also wants to enter options associated with that paragraph. Similarly, when deleting an unwanted paragraph, the program will offer to delete options associated with that paragraph (where startpararef value in options matches pararef value in paragraph).  Also if the primary key (pararef) value of a paragraph record is changed, the program offers to update the startpararef value of options associated with the paragraph.  These functions are also adapted to the situation with monsters and associated equipment.

This customised version is a lot longer at 665 lines - as well as writing functions that can cope with any table, I have also written functions to deal with specific tables. Is it better? When I am using it to add data to the database, yes I believe so. But I have to hope that I won't be changing the SQLite table structure because the second version assumes the database is set up in a particular way.
Incidentally, both versions can present the contents of each table in HTML. The first one can do so in a straightforward, no-frills way.

Current Contents of paragraph

modulepararefparatextmonsterlooseitem
haunted001The storm is raging, and you are pelted by rain and hail. You take shelter in the porch of an old creaking house. Perhaps you can find food and a bed for the night? You knock on the door, but it is not locked and swings open. You consider going in or looking around the outside of the house.
haunted001aYou find yourself at the front of the house. The front door is still ajar. There are stretches of garden either side to the north and south.
haunted002You walk around to the north side of the house. The windows on the ground floor are all boarded up and you cannot see in. Through the pelting rain you can see what may be a vegetable garden further to the north. The garden continues along the side of the house.
haunted003Walking around the south side of the house you come across a large pond. In the middle is a statue of what may be a satyr or a goatish demon. The water in the pond looks murky with pondweed. The south side of the house continues ahead.
haunted003aAs you step into the cold murky water the mud covers your ankles. You move in, towards the mysterious statue. You suddenly see from the corner of your eye something moving in the water towards you. Something dark green, slimy and with long limbs lurches at you from under the water. You must defend yourself against the giant frog!Giant frog
haunted005You approach the vegetable patch, and notice some figure shuffling around behind the canes supporting runner beans. Covered in rags and wielding a pitchfork it lurches towards you. As you see its rotting face, you realize the gardener is an undead zombie and you must fight it!Zombie
haunted005aYou approach the vegetable patch. The remains of a defeated zombie are scattered over the cabbages and broccoli. Just to the west you can see a surprisingly large garden shed.

The second one has the option of including the options associated with each paragraph:

Current Contents of Paragraph & Options table

modulepararefparatextmonsterlooseitemoptions
haunted 001 The storm is raging, and you are pelted by rain and hail. You take shelter in the porch of an old creaking house. Perhaps you can find food and a bed for the night? You knock on the door, but it is not locked and swings open. You consider going in or looking around the outside of the house. ('Do you go left around the north side of the house?', '', '002')
('Do you go right around the south outside of the house?', '', '003')
('Do you step into the dark house?', '', '004')
haunted 001a You find yourself at the front of the house. The front door is still ajar. There are stretches of garden either side to the north and south. ('Head left to the north side of the house?', '', '002')
('Head right to the south side of the house?', '', '003')
('Push open the door and go into the house?', '', '004')
haunted 002 You walk around to the north side of the house. The windows on the ground floor are all boarded up and you cannot see in. Through the pelting rain you can see what may be a vegetable garden further to the north. The garden continues along the side of the house. ('Go over to the gardening figure in the vegetable patch?', 'Alive:Zombie', '005')
('Carry on along the north side of the house?', '', '006')
('Head back east to the front door?', '', '001a')
('Head over to the vegetable patch to the north?', 'Dead:Zombie', '005a')
haunted 003 Walking around the south side of the house you come across a large pond. In the middle is a statue of what may be a satyr or a goatish demon. The water in the pond looks murky with pondweed. The south side of the house continues ahead. ('Wade into the pond to investigate the statue?', 'Alive:Killer frog', '003a')
('Wade into the pond to investigate the statue?', 'Dead:Killer frog', '003b')
('Head back east round to the front door?', '', '001a')
haunted 003a As you step into the cold murky water the mud covers your ankles. You move in, towards the mysterious statue. You suddenly see from the corner of your eye something moving in the water towards you. Something dark green, slimy and with long limbs lurches at you from under the water. You must defend yourself against the giant frog! Giant frog
haunted 005 You approach the vegetable patch, and notice some figure shuffling around behind the canes supporting runner beans. Covered in rags and wielding a pitchfork it lurches towards you. As you see its rotting face, you realize the gardener is an undead zombie and you must fight it! Zombie
haunted 005a You approach the vegetable patch. The remains of a defeated zombie are scattered over the cabbages and broccoli. Just to the west you can see a surprisingly large garden shed.
This gives a better idea of how the records in different tables relate to each other.

Sunday, 9 December 2018

A Script to Look at Database Files

This program arose to make things easy for me in the long run, a kind of constructive laziness (as discussed here). There are times when I am writing programs that use database files, and I forget what the table names are, or what the column names and types are, and SQLite needs these to be correctly spelt (and is case sensitive). I could open up sqlite3 on the Windows command line and try to remember how to view the metadata about the database, but that can be a bit of hassle, particularly when my mind is focused on Python code.
This program looks at that metadata and presents it in a relatively easy way.

import sys
import os
import webbrowser
import sqlite3

# Collecting list of database files
dirlist = os.listdir('C:\\sqlite\\db')
dbfilelist = []
filecount = 0
for file in dirlist:
    if file[-3:] == '.db':
        print(filecount, file, 'found')
        dbfilelist.append(file)
        filecount +=1
print(filecount, 'Completely different directory')
# Selecting which db file
filechoice = input("Enter number of file to view: ")
try:
    fileno = int(filechoice)
    chosenfile = dbfilelist[fileno]
    print(chosenfile, 'selected')
    dirpath = 'C:\\sqlite\\db\\'+chosenfile
except:
    try:
        if int(filechoice) == filecount:
            dirpath = input("Please enter new directory path including filename: ")
            # Check that new dirpath is valid file
            fcheck = open(dirpath, 'r')
            fcheck.close()
            chosenfile = dirpath
    except:
        print("Sorry, cannot parse that.")
        sys.exit()

# Starting up SQLite and selecting tables
conn  = sqlite3.connect(dirpath)
curs = conn.cursor()
curs.execute('SELECT name FROM sqlite_master WHERE type="table";')
tabdump = curs.fetchall()
tabcount = 0
print("Current tables in", chosenfile)
if len(tabdump) == 0:
    print("Database seems to be empty - quitting")
    sys.exit()
for table in tabdump:
    print(tabcount, table[0])
    tabcount += 1
tabchoice = input("Select which table to inspect or Q to quit: ").lower()
if tabchoice == 'q':
    print("Quitting - bye!")
    sys.exit()
else:
    tabname = tabdump[int(tabchoice)][0]
    print(tabname, "table info")
    curs.execute("PRAGMA table_info("+tabname+");")
    pragdump = curs.fetchall()
    collist = []
    for line in pragdump:
        print(line)
        collist.append(line[1])
    curs.execute("SELECT * FROM "+tabname+";")
    contentsdump = curs.fetchall()
    rows = len(contentsdump)
    print(tabname, "contains ", rows, "records/rows")
# Creating HTML table   
if rows > 0 and input("View table as HTML (Y)? ").lower() == 'y':
    FH = open('temp.html', 'w')
    FH.write(" \n")
    FH.write("<HTML> <BODY>\n")
    FH.write("<TABLE Border=1> <TR>")
    for col in collist:
        FH.write("<TD><B>"+col+"</B></TD>")
    for line in contentsdump:
        FH.write("<TR> \n")
        for cell in line:
            FH.write("<TD>"+str(cell)+"</TD>")
        FH.write("</TR> \n")
    FH.write("</TABLE> \n")
    FH.write("</BODY> </HTML> \n")
    FH.close()
    webbrowser.open('temp.html')
   
conn.close()



This script works well and makes relatively few assumptions. As a matter of good habit, I create and keep database files in 'C:\SQLite\db' folder in Windows, and I always give them a '.db' extension at the end of the filename. This script assumes the user wants that folder, but the user can go for a completely different directory path. This option also allows for users to try to open files without the .db extension.  The os module proved useful in listing files.
The sys module was used to exit gracefully in a number of situations where the program could not progress.
The tables within each database can be found by looking at the sqlite_master table, which holds other information as well.
The SQLite PRAGMA command was useful in getting information about a table in the database, including the columns in the table.
Finally there is the option of viewing an HTML version of the table, using a similar method to that used before in my previous programs.
Typical results are shown here:

= RESTART: C:/Users/666/Dropbox/Misc Programming/Python/python3/db_viewer.py =
0 base1.db found
1 biology.db found
2 dragon.db found
3 example.db found
4 scimitar.db found
5 Completely different directory
Enter number of file to view: 0
base1.db selected
Current tables in base1.db
0 MonstrousTroops
1 loot
Select which table to inspect or Q to quit: 0
MonstrousTroops table info
(0, 'Name', 'TEXT', 0, None, 0)
(1, 'Challenge', 'REAL', 0, None, 0)
(2, 'Army', 'TEXT', 0, None, 0)
(3, 'Notes', 'TEXT', 0, None, 0)
MonstrousTroops contains  29 records/rows
View table as HTML (Y)? n


Tuesday, 4 December 2018

Historical Database ver5 - combining SQLite, Python and HTML

I have had a go at updating the program for looking at and managing the database of historical figures. Last post I was busy altering the SQLite database to add a new column (imagelink, for images associated with each record) and remove an unwanted column (Role, which has been replaced by a subtable to allow multiple roles per personality).

Now I have been modifying the Python program firstly to accommodate the changes in the data structure, and also to display any selected record as a HTML page. This is for two reasons:
Firstly HTML pages can display images when opened in a browser; Secondly HTML can take web links in the database and turn them into clickable hyperlinks. You can't do either of these just on the IDLE command line.(A third reason that isn't put into practice here is HTML makes tables easier, particularly automating column width - useful when you want to display data in tabular form).

 The whole program is now 415 lines long, so I won't just copy and paste the whole thing. That's one of the things I've noticed since starting out learning Python. When you keep working on a program, it can grow surprisingly big. The trick is to make sure that big does not mean unreadable or unmanageable (functions really help here, as do comments in the code).
Making changes to the data structure can mean lots of changes to Python code, and I've been going through the code and testing it, seeing what the ramifications of changing the columns are for the code. This has affected editing existing records, entering new records, displaying the main table and of course displaying records.

This is the function that has changed the most, the one for displaying individual records.

def displayrecord(name):
    curs.execute("SELECT * FROM historicalfigures WHERE Name = '"+ name +"';")
    maindump = curs.fetchall()
    curs.execute("SELECT * FROM roles WHERE Name = '"+ name +"';")
    rolesdump = curs.fetchall()
    curs.execute("SELECT * FROM weblinks WHERE Name = '"+ name +"';")
    webdump = curs.fetchall()
    mainrecord = maindump[0]
    FileH = open("temp.html", 'w')
    FileH.write("<HTML>")
    FileH.write("<h3>Name: " + name +"</h3> \n")
    FileH.write("<p>Born: " + str(mainrecord[1])+"</p> \n")
    FileH.write("<p>Died: " + str(mainrecord[2])+"</p> \n")
    FileH.write("<p>Nation: " + mainrecord[3]+"</p> \n")
    FileH.write("<p>Image: " + str(mainrecord[4])+"</p> \n")
    FileH.write("<br><img src='"+str(mainrecord[4])+"' width=300 align='right'>")
    rolestring = "<p><B>Roles</B></p> <ul>\n"
    for line in rolesdump:
        rolestring = rolestring + "<li>"+ line[1] +"</li> \n"
    rolestring += "</ul> \n"
    FileH.write(rolestring)
    if webdump: # does webdump contain any lines?
        FileH.write("<p><b>Web addresses for more info</b></p> \n")
        FileH.write("<ul>")
        for line in webdump:
            FileH.write("<li><a href ='"+line[1]+"'>"+line[2] +"</a></li> \n")
        FileH.write("</ul>")
    FileH.write("</HTML>")
    FileH.close()
    webbrowser.open("temp.html")



As you can see it first collects the data from SQLite (using the curs cursor object as usual), then creates a file (temp.html) and file handle (a.k.a. io stream) so Python can write to it, and then writes HTML code to the file modified by what the data in the database is. Finally it uses the webbrowser module (imported along with sqlite3 module earlier in the code) to open the newly created web page in your web browser.

Here is the HTML contents of a typical record:

<HTML><h3>Name: Cardinal Thomas Wolsey</h3>
<p>Born: 1473</p>
<p>Died: 1530</p>
<p>Nation: England</p>
<p>Image: https://collectionimages.npg.org.uk/large/mw06903/Thomas-Wolsey.jpg</p>
<br><img src='https://collectionimages.npg.org.uk/large/mw06903/Thomas-Wolsey.jpg' width=300 align='right'><p><B>Roles</B></p> <ul>
<li>Politician</li>
<li>Cleric</li>
</ul>
</HTML>

Any web designer will tell you this is not exactly attractive, but I believe it serves as proof of concept - we can use Python to get data from a database and create a custom webpage using that data.

Saturday, 1 December 2018

Historical Databases, Copying Tables and Adding Columns to SQLite

So I have decided to go back to an older project, the historical database I used to demonstrate the basics of SQLite.
I have decided the next version should have images associated with each record. At the moment the database does not have this, so how do we modify the columns of the database?

There is the SQLite command ALTER. Because I tend to leave the data structure (table columns and how tables relate to each other) alone in Python, this is going to be a one-off, done on the SQLite command line not as part of a Python script.

Microsoft Windows [Version 10.0.17134.407]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Users\pc>cd C:\sqlite
C:\sqlite>sqlite3
SQLite version 3.20.1 2017-08-24 16:21:36
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open db/history1.db
sqlite> .tables
historicalfigures  roles              weblinks
sqlite> ALTER TABLE historicalfigures ADD COLUMN imagelink text;
SQLite>

sqlite> .mode columns
sqlite> .headers on
sqlite> SELECT * FROM historicalfigures;
Name               YearBirth   YearDeath   Role        Nation      imagelink
-----------------  ----------  ----------  ----------  ----------  ----------
Leonardo da Vinci  1452        1519        Artist      Italy
Niccolo Machiavel  1469        1527        Politician  Italy (Flo
Cardinal Thomas W  1473        1530        Churchman   England
William Caxton     1422        1491        Publisher   England
William Shakespea  1564        1616        Author & P  England
Francis Drake      1540        1596        Explorer &  England
Henry VIII         1491        1547        Monarch     England
Elizabeth I        1533        1603        Monarch     England
Christopher Colum  1451        1509        Explorer    Spain & It
Ferdinand Magella  1480        1521        Explorer    Portugal
Dante Alighieri    1265        1321        Author & P  Italy (Flo
Michelangelo di L  1475        1564        Artist      Italy (Flo
Henry VII          1457        1509        Monarch     England
Hans Holbein       1497        1543        Artist      Germany &
Anne Boleyn        1501        1536        Queen Cons  England
Geoffrey Chaucer   1343        1400        Poet & Aut  England
Marco Polo         1254        1324        Explorer &  Italy (Ven
Catherine of Arag  1485        1536        Queen Cons  Spain & En
Edward VI          1537        1553        Monarch     England
Galileo Galilei    1564        1642                    Italy (Tus
Johannes Kepler    1571        1630                    Germany
Jane Grey          1536        1554                    England
Martin Luther      1483        1546                    Germany
Nicolaus Copernic  1473        1543                    Poland
Tomas de Torquema  1420        1498                    Spain
Donald Duck        1910        2000                    USA
sqlite>


This shows that the new column imagelink has been created (clearly no data yet). But it also reminds me that we have a redundant column - Role column has been superseded by the roles table.
How do we delete this column? Firstly I need to be sure that deleting it is the right thing to do. Deleting columns containing data should not be undertaken lightly.
I have discovered a way to create copies of tables in SQLite3, so I can use these to create backup tables. It involves the CREATE command, but rather than stating all the columns and their data types (which would create an empty table) I tell SQLite3 to create using an existing table (which both uses the columns with names and data types, and also uses the data contained within the table) using the format:
CREATE TABLE newtablename AS SELECT requiredcolumns from existingtablename (parameters)

 sqlite> CREATE TABLE old_historicalfigures AS SELECT * from historicalfigures;
sqlite> .tables
historicalfigures      roles
old_historicalfigures  weblinks
SQLite>


Just for reference I also found a way to rename tables.

sqlite> ALTER TABLE historicalfigures RENAME TO old_historicalfigures;
But I digress. We now have backed up our table, so can mess around with it with less trepidation.
I have now found out there is no easy way to actually delete or drop an unwanted column in SQLite. Other forms of SQL will allow this. The best workaround is to recreate the table without the unwanted column. Actually this is not too bad - we've already discovered how to do this.

To delete a whole table
sqlite> DROP TABLE historicalfigures;
sqlite> .tables
old_historicalfigures  roles                  weblinks
SQLite>

Now we recreate the table we want with the columns we want.
sqlite> CREATE TABLE historicalfigures AS SELECT Name, YearBirth, YearDeath, Nation, Imagelink FROM old_historicalfigures;
sqlite> SELECT * FROM historicalfigures;
Name               YearBirth   YearDeath   Nation      imagelink
-----------------  ----------  ----------  ----------  ----------
Leonardo da Vinci  1452        1519        Italy
Niccolo Machiavel  1469        1527        Italy (Flo
Cardinal Thomas W  1473        1530        England
William Caxton     1422        1491        England
William Shakespea  1564        1616        England
Francis Drake      1540        1596        England
Henry VIII         1491        1547        England
Elizabeth I        1533        1603        England
Christopher Colum  1451        1509        Spain & It
Ferdinand Magella  1480        1521        Portugal
Dante Alighieri    1265        1321        Italy (Flo
Michelangelo di L  1475        1564        Italy (Flo
Henry VII          1457        1509        England
Hans Holbein       1497        1543        Germany &
Anne Boleyn        1501        1536        England
Geoffrey Chaucer   1343        1400        England
Marco Polo         1254        1324        Italy (Ven
Catherine of Arag  1485        1536        Spain & En
Edward VI          1537        1553        England
Galileo Galilei    1564        1642        Italy (Tus
Johannes Kepler    1571        1630        Germany
Jane Grey          1536        1554        England
Martin Luther      1483        1546        Germany
Nicolaus Copernic  1473        1543        Poland
Tomas de Torquema  1420        1498        Spain
Donald Duck        1910        2000        USA
sqlite>
sqlite>


And so we have added a new column, and not so much removed an unwanted column as recreated the table without the unwanted column.
Next time I hope to tie this in with Python programming.