Bob Browning's blog

My previous blog post outlines my success in converting a data structure definition from one format to another. Both ChatGTP and Gemini had a really good stab at it. The result was mainly accurate and extra information had been added for me. ChatGTP managed to make a mistake on the second attempt at the task, and when I asked it why, it used a completely made-up (and incorrect) rule about SQL Databases as an excuse. So Turing Test passed!

I gave it another task. Convert a routine from using the SQLite3 data base management system, to the higher performance MySQL. This is a straightforward, if tedious task. How did it do?

The tool that I used was GitHub Copilot. It gives you a choice ot two AI ‘engines’, ChatGTP and Claude 3.5 Sonnet. I selected ChatGTP for the first attempt.

Initially it did silly thinks like deleting all my JSDOC¹ comments, plus removing code from at least one of the functions. I added items to the requirement so that I covered all of the issues. The final request was this :

Please convert the code in MySQL.ts to access the a MySQL database. The code there is currently using sqlite3. 
Note that sqlite3 doesn't need a password to connect but MySQL does. The password is in auth.mysql.user and auth.mysql.password  auth is defined in the code.
Please only make the changes essential for the conversion. If you see faults or improvements in the existing code please let me know but don't make the changes until I have reviewed them.  
Please retain the comments. Fix any errors or omissions in the JSDOC comments please. 
Use CommonJS for compatibility reasons.
Use async/await.

The ‘please only make essential changes’ was ignored. In one function large chunks of code were still removed. (Turing test failed – a programmer would not have bothered to rework code.) I asked it why and it reran the conversion with the exact same results. It seems to have interpreted the removed code as non-essential and in some systems they would be, but in my case it was essential.

Claude 3.5 Sonnet made a much better fist of the job. Most of the revised code looked correct except for this (rather inelegant for historical reasons, but it worked) section. It is loading modules from various directories on the system:

const sqlite = require('sqlite');
const sqlite3 = require('sqlite3');
import { open } from 'sqlite'
const auth = require('../../../local/auth')
const trace = require('track-n-trace')
const suds = require('../../../config/suds')
const local = require('../../../local/suds')
const tableDataFunction = require('../table-data')
const sqlFunctions = require('./sql-functions');
const mergeAttributes = require('../merge-attributes')
const lang = require('../../../config/language').EN
import { Properties } from "../../types-schema";

Was turned into this:

const mysql = require('mysql2/promise')
const trace = require('../trace')
const sqlFunctions = require('../sql')
const { suds } = require('../state')
const { auth } = require('../auth')
const { mergeAttributes } = require('../attributes')
type Properties = {
  [key: string]: any
}

So many issues!

• The sqlite lines had to go and replaced by mysql2. Full markes
• All the other lines in this section were changed. None of them were relevant to the task at hand
• Paths were changed for no good reason.
• There is no ‘state’ module anywhere in the system.
• The Properties type is complex and was imported from an external file. This was removed and a generic type added.

Everything else looked reasonable, but I didn’t check it line by line. At this point I realised it would be easier to make the changes manually. However I used the AI generated code as a guide which made the process much quicker.

At this point I realised that our attitude towards computers had to radically change. For the past 70 years we have assumed that any erroneous results from a computer were down to human error. Either bad data or bad coding². But now we had to reckon on computer error.

The idea that you can sack your programmers and use AI to generate code is for the birds. However GitHub Copilot is a great tool that sits right in to the editor and allows you to ask technical questions, review legacy code, create code and so on. Sometimes you spend hours trying to debug a piece of code, that Copilot will suggest fixes in seconds. It is a very significant productivity aid as long as we learn to use it properly – whatever that is. It is going to be a leaning process for sure.

1. A standard for documentation in programs. ↩︎
2. The technical term is Garbage in, Garbage out (GIGO) ↩︎