Figure One – Open AI robot demo

AS I mentioned in my last blog post, I’ve started to do some work using Generative AI. One of the great things about this is that it’s one of those rare areas of IT where people in the “outside world” are actually aware of AI and what it can potentially do.

People are now sending me examples of clever uses of AI and I have to say that this is the most impressive so far….

In the demo a human has a conversation with a humanoid robot called “Figure One” developed by OpenAI.

The human asked Figure One what it could see in the environment, and Figure One described seeing a red apple on a plate, a drying rack with cups and plates, and the human standing nearby. When the human asked for something to eat, Figure One gave them the red apple, explaining it was the only edible item available on the table.

Figure One then asked the human where the dirty dishes on the table should go next, and the human correctly deduced they should go in the drying rack. When instructed, Figure One picked up the dishes and placed them in the drying rack.

This demonstration is impressive for a few key reasons:

  1. Environmental awareness: The robot showed it could visually perceive and describe the objects and layout of its environment, including identifying edible items versus dishes etc. This spatial awareness and object recognition is a fundamental requirement for any AI system operating in the real world.
  2. Task reasoning beyond simple perception: The robot demonstrated basic reasoning about the appropriate actions to take given the context. It inferred that the human wanted food, so it provided the apple. It also correctly deduced that used dishes belong in the drying rack after the human prompted it.
  3. Instruction following: The robot could understand and follow the human’s spoken instructions, like “put them in the drying rack.” This ability to comprehend natural language commands and translate them into physical actions can be difficult for AI systems.
  4. Dexterous manipulation: To successfully pick up the dishes and place them in the rack required visual/mechanical coordination and dexterity.

While this is a simple scenario, Figure One exhibited several core competencies that AI and robotics researchers have been working towards for decades – perception, reasoning, language understanding, and dexterity. Integrating all these capabilities into one system that can interact in an environment is an impressive feat of artificial intelligence. It suggests progress towards more capable household/service robots that can understand their surroundings and assist with tasks based on dialogue.

My first “Hackathon” and AI

It’s some time since I added a blog post to this site. Since my last article (3 years ago), my role has changed and I’ve been less of a hands-on tester than before. I’m still involved in helping Software, DevOps and Quality Engineers, evaluating new products and trying to find ways to make their lives easier investigating tool integrations and more recently AI.

AI seems to be on everybody’s lips so I was keen to separate the myth from reality when I took part in a hackathon hosted by Google with a team from Lloyds Banking Group’s Quality Engineering Centre of Excellence (QECoE). The hackathon focused on exploring ways that generative AI could be applied to help in the workplace.

Our team at the Hackathon

Our team’s challenge was to develop an application that could enhance quality assurance processes by assessing and providing feedback on testing documents compared to “golden standards”. We created separate golden standards for different types of test plans like accessibility, functional, and performance testing. The application we built used Python to make API calls to Google’s Gemini Pro large language model, with a Streamlit web frontend. When a user uploads a document, it is compared with the relevant golden standard checklist and feedback is provided.

Since many of us weren’t experienced coders, we used AI assistants like ChatGPT and Claude for help with coding the application. Their code completion capabilities were amazing! After the hackathon, we added chat functionality so the app could interactively guide users on improving their documents based on feedback.

Our innovative solution allowing live assessment of document quality against configurable golden standards won in the “documentation” use case category. Presenting it live to the judges was a great experience.

You can read more detail here:
LinkedIn: Producing “Gold Standard” Documentation using AI

Peak Automation

During a walk with friends yesterday, we got onto the topic of DevOps automation (don’t worry, I don’t bore all my friends with tales of CI/CD). This particular friend does work in IT consultancy, so we have a common interest.

I was talking about how code commits can trigger automated deployments, environment provisioning, regression tests and so on and (if you’re feeling brave) even deploy to production.

Today this friend sent me a link to this video. I must have explained automation well, this is spot on!