Random thoughts on CSV files

Why bother with SQLite when I could use CSV files?
Since restarting this blog a month ago all of my posts have involved SQLite to varying degrees. For those who are expecting to just learn about Python this may be an annoying side-trek. One alternative I discussed in an earlier post is using CSV or text files to hold persistent data. Python can open, read from and write to these files. There are a number of advantages that SQLite has.

• SQLite is faster when file sizes and quantities of data increase. In terms of speed, SQLite is something of a compromise. For enterprise/corporation level databases, something like Oracle or Microsoft SQL Server are in a whole different league of performance and can cope with millions of records and thousands of tables (see this page for MS SQL Server). Nonetheless, SQLite is better than using Python manipulating CSV files, and it is easier for me to install and run than the enterprise-level SQL programs. 
• SQLite does a lot of work that you would otherwise need to do in Python. This is something I've appreciated even at this relatively low level. One well-written SQLite command passed by the cursor object can replace a dozen lines of Python code with their associated possible errors. As stated before, SQL is a language itself, and brings with it extra capabilities for managing and searching databases, especially relational databases. The effort put into learning the basics of SQL starts to pay off quickly. 
• SQLite is professionally more useful, mainly because it uses SQL and a lot of businesses use SQL-based programs and databases rather than CSV files. They might use a different SQL program (like those mentioned above) but the SQL is mostly the same.  
• Editing data in SQLite is far easier than reading/writing to CSV files, particularly if there is one or two items of data to edit in the middle of the table but other data around it does not need to be changed. For SQLite you can use the UPDATE command. For CSV files it is not easy to rewrite a specific line while leaving the rest of the file alone. At my (limited) skill I would end up reading the whole file into Python, making modifications in Python, then overwriting the entire file with the new version held in Python. Although doable with small files, when you are dealing with big files this can become very inefficient. 

Having said all that there are modules that help Python to interact with CSV files, including the csv module documented here. I probably ought to learn these, but right now SQLite seems better, particularly now I've made the initial investment of time and effort in getting SQLite running in my Python programs. CSV files can also be opened and edited in spreadsheets, unlike SQLite files, thus partially nullifying my final bullet point.
And SQLite, being a language with many nice features, can import data from and export data to CSV files, so if you change your mind after entering big tables of data you can always move your data from one to the other.

Creating web pages using Python

Much of today's computer usage that I am aware of occurs on the World Wide Web. Social media, news websites, forums, wiki pages, homepages for movie streaming, internet shopping and auctioning, the list goes on.
Can you create web pages with Python? Yes.
Let me qualify that.  Python can help automate and speed up writing the HTML. However, on its own it is not a web hosting service, so using it to create web pages that the rest of the world can visit will require more than just Python.

In terms of viewing web pages and opening up web browsers, the module I have found useful is webbrowser, which can be imported like other modules. For example, creating a very simple web page in Python then opening it in a browser:

import webbrowser
fhandle = open('temp.html', 'w')
fhandle.write("<!DOCTYPE html> \n")
fhandle.write("<html> \n")
fhandle.write("<body> \n")
fhandle.write("<H1>Hello World Wide Web!</H1> \n")
fhandle.write("</body> \n")
fhandle.write("</html> \n")
fhandle.close()
webbrowser.open_new_tab('temp.html')

HTML (Hyper Text Mark-up Language) is of course the basic language of the web, and many pages are created using it (especially the older ones). Like Python it is a computer language, but while Python is a programming language and SQLite is a query language, HTML is a mark-up language, which basically tells the web browser on your computer how to display the content downloaded as the web page. I won't go into the details of writing HTML, but there are plenty of books and web pages to help you. Here we are creating the HTML as strings and writing it into the new file (here temp.html).

You can also use the webbrowser module to open pages hosted elsewhere if you know the web address. For example:

import webbrowser
webpage = "https://www.bbc.co.uk/news"
webbrowser.open(webpage)
webpagelist = ["https://apod.nasa.gov/apod/astropix.html",
               "https://en.wikipedia.org/wiki/Main_Page",
               "https://www.youtube.com/?hl=en-GB&gl=GB"]
for page in webpagelist:
    webbrowser.open(page)

The webbrowser module has several functions, including
webbrowser.open('web address') 
This is the basic function that opens the web page in the default browser.

webbrowser.open_new('web address')
This function opens the page in a new window in the default browser.

webbrowser.open_new_tab('web address')
And this one opens the page in a new tab.
You don't really need Python to open up a web page, and it's a programming language, not a search engine. So why bother?
The real benefit of Python is in creating new web pages where the content is contingent on inputs, i.e. web pages that have variable content. I have already used Python to create HTML tables. This module allows you to create and display the table in the same program. So if we have a text file with the following content:
Aardonyx, Africa, 195, Sauropod-prosauropod, https://en.wikipedia.org/wiki/Aardonyx
Abelisaurus, S-America, 80, Therapod-abelisaur, https://en.wikipedia.org/wiki/Abelisaurus
Abrictosaurus, Africa, 200, Ornithopod, https://en.wikipedia.org/wiki/Abrictosaurus

Abydosaurus, N-America, 105, Sauropod, https://en.wikipedia.org/wiki/Abydosaurus 

..
.. 
.. 

we can create an HTML table using the contents and even turn the web addresses for each line into hyperlinks. For those who are wondering, this text file is from a previous project from before I started this blog, with 150 dinosaur genera.
Here is the program that does that.
import webbrowser
fhandle = open('dinodata2.txt', 'r')
totallist = []
for line in fhandle:
    linelist = line.split(',')
    linelist[4].rstrip()
    totallist.append(linelist)
fhandle.close()
xhandle = open('tempweb.html', 'w')
xhandle.write("""
<!DOCTYPE html>
<html>
<head>
<title>Dinosaur List</title>
</head>
<body>
<h1>Dinosaur List</h1>
<table style="border: 1px solid black;">
<tr><td>Genus Name</td><td>Continent</td><td>Millions of Years Ago</td><td>Group</td><td>Wiki Link</td></tr>
""")
for line in totallist:
    xhandle.write("<tr>")
    for cell in line:
        if cell == line[4]:
            xhandle.write("<td style='border: 1px solid black;'><a href='"+cell+"'>Wikipedia Link</a></td>")
        else:
            xhandle.write("<td style='border: 1px solid black;'>" +cell+"</td>")
    xhandle.write("</tr>")
xhandle.write("""
</table>
</body>
</html>""")
xhandle.close()
webbrowser.open('tempweb.html')

The results are basic - someone who knows either HTML or CSS (Cascading Style Sheet) formatting could add more style, but the important thing is it reads from a text file and creates a web page with a table, filled with data from the text file, and hyperlinks for web addresses (here the relevant Wikipedia page). It actually stores the content from the text file as a list of lists (here totallist), then uses that list of lists to create the html table. I guess the next step would to be add sorting and filters to this table so you can look for dinosaurs according to parameters.
The final line webbrowser.open('tempweb.html') shows that the webpage can be local rather than on a remote server - the web address can be replaced by a file path on the local drive or network. Here the file tempweb.html is created in the same folder as the Python program. 

So why did I choose Python? What about other Programming Languages?

The short answer is: Python seems to be the best compromise between power and simplicity.
There is also the advantage that, at my current stage of learning, Object-Oriented Programming in Python is optional.

Choosing and championing programming languages can seem like religion rather than computer science. There are proselytisers, explaining why their language is clearly the best, and other languages are fatally flawed. Practitioners have faith in their language, and have scepticism, if not outright prejudice, towards others. My experience of other languages is both small and shallow, so I am certainly not claiming any expertise in this hotly contested subject, so anything I say about each language below is purely from my own viewpoint. But I like to think of myself as open-minded and ecumenical.

Java

I don't think I would have got into Java on my own - it was actually a requirement of my Open University course.
The good: It is a powerful, professional language. A lot of off-the-shelf software is written in Java, including my favourite, Minecraft. Anything that helped create Minecraft must be good.
And yes, I did get into it for a while. I could write simple programs as required by the course.
The bad: Everything is required to be a class, as per OOP. This makes the archetypal "Hello World" program a lot longer and more complicated than in simpler languages. For a noob like me this is quite off-putting.
It's a compiled language, rather than an interpreted language, so when you want to run programs, you have to compile it first. My approach to programming involves a lot of running, testing and debugging - a kind of messy, unstructured version of rapid prototyping - compiling just slows that down.

Perl

I have to say, it was a fairly close choice between Python and Perl for me - this blog might have been titled "Progressing Through Perl" after having a go at 'Learning Perl' by Schwartz, Phoenix & foy (O'Reilly's Llama book) . What tipped the balance was relative popularity, particularly in the job adverts. There are still fairly regular vacancies for Python developers. Requests for Perl developers are very rare.
The good: It's available on any Linux system and can easily be installed on Windows (I used Strawberry Perl).
It's simplicity is comparable to Python, at least at the levels I was at (which was noobishly simple).
It's interpreted, so testing programs is quick, allowing rapid correction of problems.
The bad: Syntax isn't great. Having said that, I don't find it as off-putting as some people do.
Another thing is that it isn't really suited to Object-Oriented Programming. I understand it can be done, but it's more of a work-around than an inbuilt feature. And despite my misgivings about Java, I recognise that if I am to be taken seriously as a programmer in the workplace, I need to do OOP.

QBasic

I know that very few take this one seriously. And I don't think I ever took it that seriously. But it is what it is - a simple language primarily intended for education and training.
My first ever computer was a Commodore 64. That had BASIC on powering up. Although I was never serious about becoming a programmer until I started the OU course, that Commodore 64 gave me my first taste of programming. So when I found I could download and run a version of BASIC, in this case QBasic, on a Windows PC, I did so. I didn't get very far, but it reminded me that I can program (if only "Hello World" at that stage). I think that was more about familiarity and maybe nostalgia than seriously trying programming.
The good: It is deliberately simple, at least to start with. Some advocates (yes there are a few out there) say that QBasic has hidden depths - I haven't got round to exploring those myself, but I'll take its supporters' word for it.
The bad: As one might expect, it was created primarily as a teaching tool rather than a professional language for commercial applications. I don't know if mentioning it on your CV is sensible or not.