Contact Us

The Ultron Project

When I was contacted by Tony Stark to work on the Ultron project, to say I was shocked is an understatement. I knew from second hand stories that he had a flare for the dramatic and that he liked to do his high profile recruitment first hand, but I wasn't expecting the visit. My first thought was “You are insane”, but after a quick overview of the project and his descriptive vision I was on board. He is also pretty cool to hang out with... Given the current interplanetary political environment it seemed prudent to get something of this caliber in place and out of bureaucratic hands since those can be co-opted by less than scientific minds. (cough)

Ultron Poor Choices for Posture Pre Load

The Ultron project was a mess when I was initially brought in. It looked like the engineering equivalent of a scatterbrained genius child and was unbelievable dis-organized. I knew I had a major organizational and technical nightmare to unravel and get on track. On top of that the time lines that I was given were impossible given known earth technologies and historical precident of similar projects. Luckily we were given virtually unlimited resources and access to the alien technologies that had been acquired by Mr. Stark's side projects.

Had I known at the time where this would end with Sokovia I would have blown the whistle sooner. My condolences to the people of Sokovia and if only Mr Stark had listened to me more. I remember the feelings that day as being on par with 9/11 but better more eloquent pieces have been written on that.

The Ultron project had to be broken down into a laundry list of artificial intelligence, robotics, nanotech, Xenotech sub-projects. We brought in some of the finest minds from around the world to work on it and there were multiple changing of the guard over the course of the project.

Artificial intelligence is my specialty and while acting as program manager of the project as a whole, I headed up the AI division specifically while relegating the others to my sub-program managers.

The resources were expensive, but our budget was essentially unlimited with funding coming from governments all over the world but primarily the US government. They were OK with giving funding to Mr. Stark for the project given his previous track record and history of self sacrifice in the name of the US. Other countries knew that only the US had the resources necessary to pull of a project of this magnitude.

The organization structure was:

Keith Aumiller – Formerly of NRG and Citigroup

- Program managers, head of AI division

Steve Jurczyk – Formerly of NASA

- Head of Xeno technology R&D

Martin Buehler - Formerly of Boston Dynamics and iRobotMule Boston Dynamics

- Head of Robotics and mechanical engineering

Sally Smith - Formerly of Boeing

- Head of Nano technology

The time-line was short with only two years to provide a minimally viable product that could be used in a field test. Mr. Stark provided us access to some of his most private technologies that without, the project wouldn't have been able to get off the ground. Namely, his Neural Network technology that he had been training for years in scientific and robotics mechanics “Jarvis” as well as his ARC power reactor technology that would allow for power beyond anything anyone else in our world had access to.

Jarvis wasn't anything that special given current responsive bots built on top of googles and apples APIs, but The ARC reactor was nothing short of a physics miracle.

Most of the development was done remotely but all of it was brought home to Stark tower in New York to finalize integration and do field testing.

The initial incident with Ultron occurred after a failed field test at Stark tower after the lab had been shut down for the evening. Luckily there was some sort of private party for Mr Stark and with those friends he was able to prevent it from expending city wide.

In retrospect, experimenting with with xeno-technology in a heavily populated city was a poor decision that while I protested against, I was overruled.

While the robotics, xeno-tech, and nano-technology was absolutely amazing in it's own right, and all of us have a data science and machine learning aspect to what we do, I was tasked in my sub-program to finalize the core general intelligence of Ultron so I'll focus on those aspects of the G factor program.

Given that the timeline was so short, we utilized the technology available to us and built off of open source and proven algorithms. We did not try to incorporate theoretical just invented strategies.

As is often the case with large programs, we worked backwards from our time line. We had a deadline of two years out, so put together a rough milestone chart and list of items we would need to invent and what we would be able to scrape together from existing technologies. For the other Divisions, I deferred to my subordinates to do the same exercises in their organizational slices and we combined the time lines, adjusted for pitfalls, and monitored contingencies in a weekly meeting. Budget was rarely discussed since it was virtually unlimitted. If you needed something or someone, you went after it aggressively and got it/them on board. A few things were impossible to buy, like vibranium, but we used substitutes.


Items we had in existence for the AI sub-program of Ultron:

  • Jarvis -


  • Proven Neural Network frameworks like FCNNFCNN

    • The Framework was already in existance and we used it for a number of required initiatives that would be needed to

    • Jarvis was built on FCNN due to it's speed flexibility

    • We extended the functionality to include things like

      • Trauma learning – which incorporates high impact learning scenarios

      • Scenario training – which incorporates the kick off of a sub function to generate a “What if” play-through the NN generates it's own set of inputs to run through it's responses and learns from them

        • Similar to dreaming or day dreaming in humans

      • Shane Legg was pulled in to work as a consultant for some of the more sophisticated General intelligence questions and strategies to be implemented

    • Grzegorz Klima was brought on staff to consult on the best way to implement some of the extended features needed for the neural net and how the integration could be achieved without loss of speed performance

  • Proven Global search algorithms like evolutionary and Swarm based approaches

    • This provided for the “Creativity” and “Out of the box” thinking

    • Using GALib for the genetic algorithm worked quite well initially but it was 20 years out of date and a lot of functionality had to be added. Matthew Hall was consulted but not heavily involved in the project.

    • The extendability of feature sets/inputs of the global search algorithms was critical for adaptability of the programs

    • The swarm based intelligence systems factored highly into the finished ultron abomination. Once combined with the xenotech, the core programming was compromised and the decentralized nature of the global search and swarm algorithms allowed ultron to overwrite and re-deseign his own target fitness functions.


I'll stop the description of the project for now, but for further questions please feel free to reach out to me. If there is interest I'll continue a historical record of the project.

Program Manager

Data Scientist and AI expert

MBA Washington University in St. Louis

BS Psychology Truman State University