Pharmex — Project Response

Final working demonstrator and walkthrough

Available at https://expertsystemdemos.net/pharmex_final/walkthrough.html and the links from it.

Technical/component breakdown and overall architecture

  1. Project-based storage of adverts and their analyses, organised as folders.
  2. Analysis is via ChatGPT with a custom prompt. ChatGPT emits analysis findings as JSON.
  3. Clicking on an advert displays both it and its analysis. The latter is displayed both as regions overlaid on the advert image, and as text on the right-hand side of the image.
  4. This text — explained in the walkthrough — contains scores, reasons for them, and recommendations. The scores concern both direct features of the adverts and matters of regulatory compliance.
  5. Fuzzy-logic rules from Pace's fuzzy-logic engine generate the latter scores from the former.
  6. My intention was to have the analyses done via the ChatGPT API. However, access to the API didn't become available in time, so in this prototype, I do it manually by invoking ChatGPT in a web browser via semi-automated Windows batch files. As their final step, these upload their analyses (as JSON) to the appropriate project folder.

Technology stack and hosting approach

  1. Code is written in JavaScript, with HTML, CSS and SVG for display.
  2. This code is in a web page served by our web server, which runs under Apache on a commercial British hosting company.
  3. The user displays their results by using a browser such as Chrome, feeding it the URL above.
  4. In the prototype, I upload adverts for analysis from a dedicated local directory using batch files. I also invoke analysis as described in the previous section.
  5. I have written server files to automate this using the ChatGPT API, and have tested most of it on a “fake AI” stub. However, due to unavailability of the API, I have not tested that.

Feature prioritisation / future-development roadmap

  1. The architecture works and makes a simple and easy-to-use system. Its main disadvantage is that it relies on an LLM, and these are known to produce wrong results. In our next stage, therefore, a critical step will be to evaluate how much of a problem it is. If it is, we shall investigate alternative approaches using non-generative linguistic models.
  2. We should also engage experts in regulatory alignment and work with them on requirements specification, to see how much my analyses and scores might need changing.

High-level Health Canada alignment considerations

See above. Step (2) there will be crucial.

UI/UX screens and relevant user flows

See the walkthrough. The interface is straightforward and has only five different states.

Mobile/responsive presentation

I designed the latest version for mobile as well as for desktop, but I overlooked uploading and testing the relevant bits of CSS. I will do so.

Mock advertisements and test data

Available in the project directories, and documented in the walkthrough.

Testing and repeatability work

See above. I ran out of time on collating and documenting the repeatability work, partly because of the lack of AI access, and partly because manual ChatGPT has become very slow. Informal checks done a few weeks ago showed that scores and document partitioning do vary between runs, but give broadly the same results.

Basic prototype-level UAT/QAT

I understood that Rafi and Zahid would be helping with this after they sent me their test adverts. However, I didn't get any response to the trial analyses I sent back. To get round this, I did extensive testing myself on the workflow and displayed results. I decided that I needed to simplify the workflow and the displays, as well as how they were explained. This led to development of the new system from the original at https://expertsystemdemos.net/pharmex_week_7/megavit_recommendations.html.

Reporting / analysis outputs

See the walkthrough. I go through them in detail.

Recommendations functionality

See the walkthrough, which also covers these.

Documentation and functionality descriptions

See the walkthrough.

Final test examples/results

Available by viewing the project images.

Known prototype limitations

  1. Potential hallucination by LLM.
  2. Knowledge base and analysis prompts are written by a non-expert, me. They need to be redone with the help of experts.
  3. UAT and QAT need to be done by experts, on adverts written by advertisers.

Project-specific source files and final handover materials ready for release following final payment

I will make these available when the payment arrives. They are on our server.

Evidence against the 10-week plan

Week 1: Dashboard

Implement a simple web-page dashboard which enables the input documents to be examined and analysed, and the analyses displayed. For the moment, I will do the analysis separately, by manually feeding documents and analysis prompts into an LLM, then manually saving the resulting JSON file. The dashboard will have a button for selecting each meaning, and will overlay its semantic heat map onto the input document, displaying the result on the analysis page.

Demo: Screenshots sequenced to show the dashboard in use.

Progress: Done.

Week 2: Web hosting

This dashboard will run locally on a laptop. Transfer it to our web server so that it can be run from any Internet-connected computer.

Demo: Screenshots sequenced to show the dashboard in use via a web browser.

Note: The transfer can be done at various points along the timeline. Exact timing may depend on server maintenance and availability of maintenance staff.

Progress: Done.

Week 3: Input handling

I now enhance the dashboard’s input handling. Provide basic facilities for sorting and viewing input documents, categorising them by name, description, date, and other features. We will not be building anything as grand as a library catalogue here, but we will be showing that the system can be equipped to help its users find their way around its data.

Demo: Screenshots sequenced to show the enhanced dashboard in use via a web browser, with various kinds of input listing.

Progress: Done.

Week 4: Output visualisation

I now turn to the outputs. As already mentioned, my analyses decompose an input document into nested regions. Each region will carry a text string or image, and is likely to have some kind of coherent pharmaceutical interpretation. I will enhance the dashboard to describe and visualise these structures. This will be useful to implementors for debugging, but also to users, in providing extra detail, and in helping them find their way around large documents. Other kinds of charting may also be added.

Demo: Screenshots sequenced to show the enhanced dashboard in use via a web browser, with various kinds of output analysis and visualisation.

Progress: Done.

Week 5: Automating analysis

So far, I will have been doing analyses manually by sitting at a browser and feeding LLM prompts into it. To make the system work, however, we need to automate this by calling the LLM from the web. This will require a small server-side component, probably implemented in Python, which submits analysis requests and prompts to our LLM, then receives and restructures the results.

Demo: Screenshots showing the API-caller system at work.

Progress: Almost done. I implemented and tested everything needed except for calling the API itself. The client still needed to provide funded API access to a suitable LLM service.

Week 6: Testing and repeatability

Testing and repeatability. Build the test corpus; repeatedly analyse selected inputs; investigate variability and obvious failure cases. My tests so far suggest that repeated LLM analyses do not always give exactly the same result, but the variation appears small enough for the purposes of the MVP. For a later stage, repeatability and calibration could become a subproject.

Demo: Screenshots showing repeated analyses of selected test inputs and examples of the resulting variation.

Progress: Partly done. Without the API access, repeatability testing was slow and error-prone. I worked out how to partially automate using Windows batch files and the clipboard, but did not have time to prepare a formal presentation. Informal testing showed that scores and document partitioning do vary between runs, but deliver broadly similar results.

Week 7: Recommendations

We shall now have a system for efficiently organising inputs and outputs, automatically analysing documents, and displaying the resulting semantic fields. The next step is to turn findings into useful recommendations. These might include making safety information more prominent, clarifying an ambiguous scientific claim, providing missing evidence, or reducing imagery or language which could give a misleading impression. Where useful, recommendations can be illustrated using prepared examples showing an original document and a revised version, allowing their analyses to be compared.

Demo: Screenshots showing findings and recommendations for selected test documents, including a before-and-after example.

Progress: Done.

Week 8: Documentation and further testing

We are now approaching the end, and can start explaining to users how to use the system. Documentation will cover basic concepts, workflow, and how to interpret output. It will be held as web pages, linked from the system’s home page.

Demo: Screenshots of the pages and nav bars. At this stage, users are invited to try the system for themselves.

Progress: See Week 9.

Week 9: Styling and branding

At this stage, we will not change the page structure significantly, but we will style the system if desired, as well as adding logos and other branding.

Demo: Screenshots, and user usage.

Progress: Although my plan said I would not change the page structure significantly, testing showed a need to -- for using on mobile, but also because the existing UI was cumbersome and did not adequately describe complex documents. So I did restyle the system and reorganised the UI, as noted earlier. A demo was made available as my final version. The clients haven't provided any information on branding or styling, so I haven't incorporated their desires, if any.

Week 10: Final integration, adjustments and delivery

Resolve outstanding issues discovered during user testing; check the end-to-end demonstration; finalise example data, documentation and presentation material.

Progress: See Week 9.