Langchain has been a good project in the area: https://github.com/hwchase17/langchain

This seems amazing. Has anyone here tried to actually use this stuff? I am earnestly trying to create a website that can code simple applications for users. Or at least highly intelligent technical users or programmers. In particular it would be great to have an alternative to relying 100% on OpenAI for code. Or if there is for example some model that can "see" web page layout and also output markup.. I may have to experiment with some of these like this one maybe https://github.com/microsoft/unilm/tree/master/markuplm

Adept.ai is rumoured to have a model that understands screens and web pages, but they are still in pre-release stage.

If you use the image as input, you can try the DONUT model [1] that directly answers questions without external OCR. It can be fine-tuned to suit your needs.

You can get a pretty decent reading of screens and text documents with Amazon Textract. They find key-value pairs pretty reliably.

[1] https://arxiv.org/abs/2111.15664

I'd like to see a project that can answer questions like StackOverflow.

Or an agent that can work on the Bash command line and can e.g. fix apt-get installation errors or network problems.

I’m working on a similar project. I’ve been wondering for a long time how ChatGPT is able to persist conversation context between prompts so these papers are a “a ha” moment for me

Check out David Shapiro's channel on YouTube. Has Python walkthroughs