The State of Applied AISurveying the landscape: present and future.Applied AI continues to accelerate, largely fueled by the maturation of tooling and infrastructure. Couple this infrastructure with a strong pool of talent and enthusiasm, readily accessible capital, and high customer willingness to adopt AI/ML and you’ve got something special. We’re turning the corner into a new decade where AI/ML will create real value for both the consumer and the enterprise at an accelerating pace.
Defining TermsApplied AI: anything to do with taking AI research from the lab to a use-case and everything in-between. From the infrastructure and tooling, to the hardware, to the deployment surfaces in industry, to the models themselves, it takes a village to get a bleeding edge advance in AI research to a use-case. One great test for maturation in our field is the time it takes for a new advance to get from paper to production. Even just a few years ago you could skim some of the major advances in the field and struggle to find real use-cases; this is quickly starting to change.
Some choice examples:
Convolutional Neural Network research enabling self-driving cars (Tesla, Cruise, Waymo, etc.)NLP like BERT and GPT-2/3 improving search and assistants (Google Search, Siri, Google Assistant, SoundHound, Clinc)Reinforcement Learning helping many companies realize the lofty promises of AI in industrial robotics (Covariant, FogHorn, Rethink)ML for fraud detection and consumer outcomes helping banks, credit cards, and lenders work to limit fraud and manage risk (Fraud departments at banks and credit cards, Verifi, Ravelin, Stripe)GANs enabling generating fresh content, realistic faces, and improve photo quality (generated.photos, rosebud.ai, RunwayML)To get from research into production takes far more than just a model. It takes a village of both research and engineering efforts in tandem to make these things work. It takes hardware, it takes scalable hosting, it takes DevOps, it takes great data science, and much more. Thankfully, more and more startups are building solutions for each building block, and big players (Uber and Netflix come to mind) are joining in as they open source more and more of their tooling.
We’ll remember the all-stars who came up with new models, but the engineers turning it all into production code, the labelers creating your next dataset, and the mob vocally opposing the next breach of security and human rights should all be remembered for their contributions to our field.
Why now?We’re seeing huge opportunities for AI use-cases popping up in all industries. As tooling and infrastructure mature, new opportunities are becoming accessible to anyone that can write a few lines of code. Both disruption of existing markets and creation of new markets are being driven by adoption.
We’ve already seen the proliferation of Machine Learning into your search engine, fraud detection on your credit card, the camera in your smartphone, and modern marketplaces. And we’re starting to see enterprise adoption as legacy companies invest in the tools and teams necessary to augment their products and processes with ML.
In this essay we’ll cover not only the ways Applied AI has enabled some of our favorite products and features in the digital world, but we’ll also explore how Applied AI is changing workflows, creating new opportunities, and freeing up labor in fields like manufacturing, construction, supply chains, and commerce. We’ll go in depth on current trends in our field, while also taking some stances on where things are going.
We can typically identify waves of innovation as enabled by some new technology or event. And in the past decade we have seen the inflection point for AI take us from a bundle of hype to real use-cases driving value across industry.
So why is now the inflection point for a new wave of value in AI/ML?
Maturation of tooling and infrastructureAccessibility to training and serving at scaleLarge-scale models as APIsContinued access to risk capital, research grants, and government interestMaturation of tooling and infrastructureAs best practices, tooling, and infrastructure start to mature, accessibility is dramatically increasing. In infrastructure and tooling an advanced team or large open source effort remain the norm. While in practical applications we are seeing successful startups built by junior engineers, budding statisticians, and entrepreneurs willing to sift through the mud to make their application work. And say hello to the flood of MBAs interested in taking part in this wave of opportunity.
Additionally, an influx of talent, better coursework and training programs, and overall massive hype behind the movement has made hiring a good Data Scientist or Machine Learning Engineer less of a mission to outer space. Because of better tooling, Data Scientists and ML Engineers can go more narrow + deep and be very effective. And most MVPs can be built with either an off-the-shelf model or using one of the beautiful and highly accessible libraries like Scikit-Learn or Keras. We can make all the jokes we want about clf.fit(), but the fact actual models with real value are getting built in just a few lines is a good thing. When senior members of a field start to gripe about all the ‘fake engineers’ and ‘fake data scientists,’ what they really mean to say is, “I’m annoyed that junior people are doing in a few hours what used to take me a few weeks.”
And access to hardware is no longer a blocker. There are plenty of free compute hours lying around for the enterprising individual. Where early MVPs may have previously required a bit of bootstrapping or help from an angel, most non-research ideas can get off the ground with the primary blocker being access to data. This is a VERY GOOD THING. It’s fair to celebrate primary blockers to training models no longer being a niche skillset or access to expensive infrastructure.
We’re seeing general consolidation in infrastructure around a handful of core products. AWS, GCP, and Azure remain the clear winners in this wave, with hardware from Nvidia and Intel dominating the data center. We’re also seeing companies pop into the space that take on more niche approaches like cleaner training + deployment (see Paperspace and FloydHub).
We’re obviously all intimately familiar with TensorFlow, PyTorch, Scikit-Learn and the other major modeling tools. Across industry we’re seeing the continued dominance of Jupyter and various clones for most modeling workflows. There’s also a clear split between more Data Science heavy workflows and ML Engineering workflows, where ML Engineers spend their time in their IDE of choice, while modelers spend more time in Jupyter and projects like Colab, Deepnote, Count, and others with their specific advantages.
And these tools remain core to the ecosystem. But perhaps the biggest enabler in the last 5 years has been around deployments and serving. Docker and Kubernetes remain core to the ecosystem, while a number of tools have jumped in with their own unique value props. Kubeflow is quickly gaining steam, while TensorFlow Serving, MLFlow, BentoML, Cortex and others vie for similar chunks of the market by trying to enable all modelers to get their model into production with minimal effort. “Deploy models in just a few lines of code” is the tagline of numerous projects. Ease of deployment is great for customer acquisition; scaling and maintaining is what keeps customers in the long-term.
This innovation was to be expected, as the average Data Scientist and less engineering heavy ML Engineer likely isn’t terribly interested in spending too much time on DevOps, container orchestration, scaling, etc. And lots of teams are skipping out on hiring much engineering talent when building their initial core team. Mileage may vary.
I tend to look at Machine Learning efforts broadly in the following ladder. In the past we were forced to build many of these rungs ourselves or neglect certain steps entirely (messy versioning, nonexistent CI, manual scaling, only maintaining when the model is clearly broken). Thankfully plenty of teams are working to simplify our lives at almost every step:
Data management, schema, dataset versioningModel definitions, training, and evaluatingSerialization, servingDeploying, CI/CD, and model versioningMonitoring and maintainingIn some cases the above efforts are very separate. In others, the same tool handles multiple steps in the process. For example, we might see a tool for serving also easily handle serialization. In other cases the library for training might be tightly integrated with serializing (pickle, joblib, dill, onnx, etc.). The interesting part of the ecosystem is how tooling is maturing to the point where you can have a full-service tool like BentoML, but you also have plenty of other options with additional customization if needed. More engineering heavy teams might not spend any time using Bento, Cortex, or other services that are intended for less technical audiences. Whereas I personally love tools like BentoML and Cortex because they save tons of time for our small team. MLOps is coming a long way.
It seems like the piece we’re missing the most in our space is monitoring and maintaining.
Christopher Samiullah very nicely summarizes this here.
Obviously this list is incredibly biased towards tools I’ve used in the past or actively am using. Some tools which aren’t ML specific are excluded. For example, Airflow is a key part of many workflows but was exempted in this case. You’ll additionally see a clear bias for the Python ecosystem, perhaps to the chagrin of some. We also exclude databases, versioning, etc. Obviously data collection and cleaning are core to our workflows, but much of this process is not new to software engineering and is covered to far more depth elsewhere than I could ever cover here.. We mostly covered tooling excluded to modelers and ML Engineers, not Data Engineers, analysts, or BI heavy Data Science workflows.
Large scale generalized models as an APILet’s talk about the hype of GPT-3. I’m perhaps less excited about the outcomes of GPT-3 than I am about the approach as a model for the rest of the industry.
It seems likely that we’re gearing up for an arms race for the biggest and best (general purpose) models. Compute at that scale isn’t realistic for smaller companies and startups. Smaller efforts will have to favor clever optimizations and research overthrowing more and more compute at problems. A combination of the two seems to be the obvious winner here, and I’m expecting a general consolidation of the leading modeling efforts around a small group of companies with massive war chests that can afford the compute and fund the research. We will then see a few dominant players serving those models which are public and don’t require highly specialized data to work. These use-cases can be consumed by all sorts of products globally. Let’s visualize this.
The potential future consolidation of Applied AI to key players.GPT-3 is a great example of where this trend is likely headed. In just a few short weeks the Open AI API is already being accessed by dozens of great use-cases.
And across the ecosystem we’re seeing similar efforts. This model for development isn’t limited to NLP. Over in vision, a handful of autonomous driving startups with a focus on the software/hardware will likely enable the incumbents who don’t wish to do their own R&D to keep up with those who do. The general enablement of a company to tap into these massive efforts without having to perform R&D is a major win. Expect to see all sorts of models offered as a service. The large-scale models will fuel the bulk of innovation, and smaller and smaller pieces of the pie will get cut out by more niche players. As models get better and better at generalizing, expect less reliance on custom modeling efforts. And those business-specific use-cases aren’t in optimizing models as much as gathering specialized application-specific datasets. Data rules everything around me.
Deployment at (legacy) corporationsMany walled gardens will be opposed to lack of security guarantees presented by semi-private APIs. There’s a massive opportunity here for companies that can optimize large models, compress models, and make growing data lakes manageable. It’s hard to believe that legacy corporations will all demand on-premises deployment of the next-generation of 175 billion+ parameter models. But don’t put it past them.
Things start to get especially interesting when we introduce PII into the mix. Don’t be surprised by companies that laugh in your face at the mention of sending their data off their internal network to some new and trendy API. Companies that can compress models and get similar results from smaller models will remain relevant as long as compute and storage remains an expense. Cost of training and serving continues to get slashed, but deployment costs can still be pretty heavy. AI companies continue to have inferior margins to traditional SaaS companies, largely for this reason.
“Anecdotally, we have seen a surprisingly consistent pattern in the financial data of AI companies, with gross margins often in the 50–60% range — well below the 60–80%+ benchmark for comparable SaaS businesses.” — a16z
Don’t underestimate small dataBig models with billions of parameters will continue to get tons of love. And massive datasets will continue to drive the hyped models. In the reality of industry, smaller models are essential in many use-cases. You’re presented with two core decisions when building in edge scenarios:
Smaller or compressed models (i.e. TensorFlow Lite)Remote connectivity to computeWe can deploy to edge devices using solutions like TensorFlow lite. And better hardware for edge and consumer devices is coming out of companies like Hailo, Kneron, and Perceive. The pace of innovation in hardware might outpace the need for small models.
When remote connectivity is an option, we can always consider performing compute off-chip, though there are plenty of blockers and common constraints like connectivity concerns and time to compute. In environments like manufacturing this may be preferable as connectivity may have higher guarantees due to the stationary nature of the process. We’re already seeing 5G factories pop up where remote control systems are getting installed. Wireless sensors communicate back to the control system wirelessly. It stands to reason remote inference will be part of this transition. There are also plenty of use-cases where we can submit our data, complete some other task, and use our results downstream. Think of manufacturing where you might take a picture of a product upstream, perform transformations, and then downstream match the quality check to the product. This obviously isn’t an option in real-time scenarios like autonomous driving.
Small data is also incredibly attractive. To perform a successful Proof of Concept we may be tolerant of a liberal amount of Type 1 Errors, depending on the industry and use-case. Sensors can also oftentimes be invasive and the less time we need to gather the data the better. An example of a company with this stuff in mind is Instrumental, looking to solve manufacturing quality problems with minimal examples.
Don’t underestimate small data!
Access to capitalRisk capital, particularly VCs remain a primary gatekeeper to the future of innovation. And thankfully the tap is wide open on funding AI businesses. Tangentially, enterprise data businesses are also getting healthy rounds, measured by both size of rounds and pure volume of rounds.
For the common builder, bootstrapping a Machine Learning business continues to become easier and easier. A rough landing page, access to GPT-3 (or any other pre-trained model), a few cloud compute credits, and a clever tweet or two will get you everything you need to build and test your proof of concept.
All that said, any halfway decent PoC will quickly enable access to VC money, so most will quickly forfeit their ambitions of bootstrapping to profitability. And for good reason. Rounds are closing quicker, an increasing number of active angels and micro funds is enabling faster movement in pre-seed and seed rounds.
Corporate VCs (Google Ventures, Salesforce Ventures, Samsung Ventures, Intel Capital, etc.) are also especially active in Applied AI and general Data Science businesses. And it makes sense. Developing this stuff in-house is hard. Corporate VCs can help the mothership find synergies with the AI startups they invest in. And some executives still view AI as a risky bet and not worth building an organization around. If they change their mind, these investments in AI startups present both a potential way to onboard new technologies they’re missing out on, but also as a healthy source of talent in an industry where talent acquisition isn’t always the easiest. Check the recent investments of a Corporate VC like Intel Capital and you’ll see AI and general enterprise data companies up and down their deal flow.
Opportunities across technologiesAdvances in vision enabled the self-driving revolution, manufacturing breakthroughs, and much more. Advances in NLP have improved search, translation, knowledge understanding, and more. And we’ve only recently started to realize the possibilities of Reinforcement Learning, the potential of GANs, and much more.
Let’s explore some of the opportunities in a technology-specific approach. Afterwards, we will explore opportunities in an industry-specific approach. It’s interesting to observe the choices startups may make to create a broad technical solution vs taking their technical solution to a specific industry.
These are by no means exhaustive lists or even scratch the surface. They should, however, serve as inspiration and give you a high level view of the landscape. We intentionally skip RNNs, autoencoders, and certain other models for brevity’s sake.
Computer Vision (CV)Key Technologies & Buzzwords: Convolutional Neural Networks, Dropout, Object Detection (classification + location)
Frontier Uses: Classification, Scene Understanding, Tracking, Motion, Estimation, Reconstruction
Dominant Industries: Automotive, Medicine, Military, Photography
Sample Companies: Cruise, Cognex
Natural Language Processing (NLP)Key Technologies & Buzzwords: GPT, BERT, DistilBERT, XLNet, RoBERTa, Transformer-XL
Frontier Uses: Speech Recognition, Text Generation, Language Understanding, Translation, Question Answering
Dominant Industries: Hard to imagine industries where NLP can’t play a role of some kind. (Though I’m not an NLP maximalist!)
Sample Companies: Open AI, HuggingFace
Reinforcement Learning (RL)Key Technologies & Buzzwords: Markov Decision Processes, Temporal Difference Learning, Monte Carlo, Deep RL, Q-Learning
Frontier Uses: Games, Markets, Controls, Scheduling
Dominant Industries (relatively unused): Robotics, Markets & Economics, Industrial Automation (primary use-case for robotics)
Sample Companies: DeepMind, Open AI, Covariant
Generative NetworksKey Technologies & Buzzwords: Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), CycleGAN, DCGAN, cGAN, StyleGAN, Generator, Discriminator, Game Theory
Frontier Uses: Photo generation, deepfakes, super resolution, image-to-image translation
Dominant Industries: Creative & media, modeling, photography, video
Sample Companies: RunwayML, Rosebud.ai, Generative.photos
Opportunities across industriesEvery industry stands to gain from Applied ML. Finance has largely tackled issues of fraud, manufacturing has solved some of the looming questions in automation that traditional controls couldn’t solve, e-commerce continues to evolve from recommendation systems. All fields are ripe for disruption. Here are some interesting use-cases and companies in sample industries.
ManufacturingKey Technologies: Computer Vision, Reinforcement Learning, Process Optimization
Use-Cases: Quality Assurance, Industrial Automation, Process Improvement, Predictive Maintenance
Sample Companies: Covariant, Instrumental, FogHorn Systems (additionally the incumbents like Siemens and Rockwell, Cognex, and others are actively investing in and performing their own R&D to play defense)
CommerceKey Technologies: Recommendation Systems, Fraud Detection, Order Matching, Personalization
Use-Cases: Quality Assurance, Industrial Automation, Process Improvement
Sample Companies: Amazon’s recommendation kingdom is the most obvious sample, massive live marketplaces like Uber optimize live matching with dynamic pricing and routing, payment processors like Stripe and Square rely on Fraud Detection
MedicineKey Technologies: Computer Vision, Sequencing, RNNs & LSTMs, Reinforcement Learning
Use-Cases: Classification of X-Rays and other imaging, Drug Discovery, Genomics, Mapping the Brain (and much more!)
Sample Companies: Insitro, Sophia Genetics, Flatiron Health, Allen Institute (non-profit),
Autonomous DrivingKey Technologies: Computer Vision, Object Detection, Semantic Segmentation/Scene Understanding
Use-Cases: Autonomous Driving
Sample Companies: Tesla, Waymo, Cruise, and many others
ConstructionKey Technologies: Computer Vision
Use-Cases: Safety, Mapping, Visualizing, Autonomy in Machinery
Sample Companies: Intsite, Kwant, Buildot
Creative & MediaKey Technologies: NLP, GANs, Computer Vision
Use-Cases: Text Generation, Video Generation, Song and Story Writing, Assistants, Speech Generation, Modeling, Deepfakes
Sample Companies: RunwayML, Rosebud, Persado
Military & State SurveillanceKey Technologies: Let’s not encourage an AI arms race.
Use-Cases: Let’s not encourage an AI arms race.
Sample Companies: Let’s not encourage an AI arms race.
EnergyKey Technologies: Computer Vision, Reinforcement Learning, Process Optimization
Use-Cases: Predictive Maintenance, Optimizing the Grid
Sample Companies: Stem, Origami, Infinite Uptime
FinanceKey Technologies: NLP, Anomaly Detection, Traditional ML
Use-Cases: Automated Banking Experiences, Fraud Detection, Personalization, Risk Management, Wealth Management, Trading
Sample Companies: Ravelin, Tala, Verifi, Suplari, every major bank and their service providers, Quantopian
Also published on Medium.
