How John Deere grew data seeds into an AI powerhouse

During CES 2022 in January, John Deere debuted a fully autonomous tractor, powered by artificial intelligence, that is ready for large-scale production.

According to a press release, the tractor has six pairs of stereo cameras that capture images and pass them through a deep neural network – that then classifies each pixel in approximately 100 milliseconds and determines if the machine continues to move or stops, depending on if an obstacle is detected.

And in March, the Iowa-based company launched See & Spray Ultimate, a precision-targeted herbicide spray technology designed by John Deere’s fully owned subsidiary Blue River Technology. Cameras and processors use computer vision and machine learning to detect weeds from crop plants. There is one camera mounted every one meter across the width of a 120-foot carbon-fiber truss-style boom, or 36 cameras scanning more than 2,100 square feet at once.

But John Deere’s status as a leader in AI innovation did not come out of nowhere. In fact, the agricultural machinery company has been planting and growing data seeds for over two decades. Over the past 10-15 years, John Deere has invested heavily on developing a data platform and machine connectivity, as well as GPS-based guidance, said Julian Sanchez, director of emerging technology at John Deere.

“Those three pieces are important to the AI conversation, because implementing real AI solutions is in large part a data game,” he said. “How do you collect the data? How do you transfer the data? How do you train the data? How do you deploy the data?”

These days, the company has been enjoying the fruit of its AI labors, with more harvests to come.

John Deere’s long journey towards AI

John Deere’s efforts in developing artificial intelligence solutions are part of a larger trend across the agricultural landscape. Spending on agricultural AI technology and solutions is predicted to grow from $1 billion in 2020 to $4 billion in 2026, according to Markets&Markets.

The company’s journey towards AI began in the mid-nineties, when a small group of innovative engineers were split off from John Deere’s product lines, such as the harvesting combine group or the tractor group, and were told to move to Des Moines, Iowa to work on a coming new wave of technology around GPS.

According to Sanchez, a GPS-based steering system, released in 1999, was a turning point for tractor accuracy at John Deere. “The economics of that accuracy are easy to understand because you overlap less,” he said. “What sold the farmers on it, though, is that they could monitor other parts of the job rather than whether they stayed in a straight line – that was the big unlock. We’ve been building on that ever since.”

Moving towards AI opportunities

The next “aha” moment, Sanchez explained, was when John Deere tagged a geospatial location to every sensor on its vehicles. “Every kind of agronomic work, whether it’s putting a seed in the ground or harvesting a plant or applying herbicides, has a sensor associated with it, so we know what is working well in the field and what is not,” he explained.

That opened up the whole idea of geospatial maps, which John Deere immediately started developing in the early 2000s. But the data transfer was clunky, Sanchez said:  “They’re recorded on the machines, and then you had to go in with a USB drive and gather all of those and take them back to the farm and upload them on a PC.”

As a result, in 2010, John Deere realized that every large agricultural vehicle out of the factory should come with a cellular-enabled telematics box. “We started removing that friction of having to move the data from the vehicle to somewhere else to make sure the data continually moves off the vehicle,” he said.

The 2010s brought the mobile and cloud revolutions, which accelerated the ability to innovate on digital tools. By 2016, Moore’s Law (the principle that the speed and capability of computers can be expected to double every two years) brought a resurgence in the opportunities of what could be done with AI. At the time, John Deere had several small teams that had already been working on robotics concepts for at least 10 years. “We had been working with some of the top robotics universities in the country,” Sanchez said. “So we could essentially pour gasoline on our evolution to build on AI.”

Building AI capabilities

In 2017, John Deere acquired machine learning company Blue River Technologies, which has become one of the key parts of the company’s innovation efforts on AI and deep learning – looking at applications for AI on machines and other domains, including construction. “That immediately doubled or tripled the number of people working on AI,” he said. “That was a pivot point.”

However, there is also a John Deere data science team, which numbers in the hundreds, that is looking at a variety of problems, he said, including “how we build models to analyze the data that has come off the machines and provide more valuable insights back to growers.”

All AI initiatives at John Deere fall under the chief technology officer’s umbrella, Sanchez said, including an organization focused on autonomy and automation solutions. “That group has the largest concentration of AI talent and includes the Blue River organization,” he added. There is also an organization that manages all of the development of the company’s digital tools – cloud, front-end mobile applications, point web solutions – with a sizable data science team. “They’re the ones curating all of the data, making sure we’re looking at all that data with the intent of generating as many possible insights for growers as possible,” he explained.

Today, John Deere is “pretty laser-focused” on a half-dozen to a dozen solutions the organization believes are most important to continue to develop and eventually deliver to market, Sanchez said. Some of them already exist, like the new autonomous tractor.

But the company’s goal goes beyond one machine. “Our goal is by 2030, we want to have a fully-autonomous production system, meaning we want an autonomous combine and sprayer and tractor planter,” he said. Today, the company offers a fully autonomous tillage solution, which is one of four steps in the production cycle that allows farmers to prepare the land before planting. Over the next eight years, Sanchez says John Deere will be able to do that for planting, spraying and harvesting.

“That’s a big deal given the labor pressures in agriculture,” he said. “For decades, there have been fewer people wanting to live in rural areas, so that’s what AI helps unlock.” He added that this commitment to AI investments comes directly from John Deere’s current CEO, who was previously in charge of the big tech area of the company. “He understood the value,” he explained.

Searching for AI-driven precision at scale

The agricultural industry has reached an “asymptote of value you can add by going bigger and faster,” Sanchez continued. “The opportunity for value has really pivoted to being very precise – you have to be able to see what you’re doing, whether you’re placing a seed in the ground, harvesting a kernel of corn or applying herbicides.”

For example, if you planted four or five corn seeds, you would want to understand something about the current moisture of the soil, because the perfect moisture would give them the best chance to emerge from the ground as a plant in as few days as possible. You would also want to analyze the quality of the soil and put the seeds in a spot where there are more nutrients. And you’d want to make sure the seeds aren’t too close to one another, because if you do, then they start competing for those nutrients. But if you put them too far apart from each other, then you’re not optimizing the little piece of ground to plant the seeds.

“Now imagine doing that at scale, when you have to plant a hundred thousand acres over the span of two weeks,” said Sanchez. “That’s why AI already has had an impact in agriculture. That’s why we see that runway of opportunity there. Agriculture has all kinds of these perfect examples that are prime for AI, as opposed to broader, more generalized applications.”

John Deere’s ‘holy grail’ AI quest

John Deere remains on a quest to tackle a couple of big ‘holy grail’ ideas around AI. One of them goes back to autonomy. “To imagine a fully autonomous production system, you have to imagine a whole system where not only can these machines do the jobs in the field, but they also can figure out what field they should move to next,” Sanchez said. “And we have to figure out how they move from field to field without significant human labor.”

The second is around the tremendous opportunity both for profitability as well as sustainability in agriculture, in terms of truly understanding the health of every inch of soil that is being used for agriculture. “So there’s a bigger game here, which is if you can farm in such a way that every year your soil gets healthier, then over time that allows you to really achieve that objective of doing more with less,” he explained.

But, he added, it’s really hard to measure things like nitrogen, potassium or sodium in real time in a reliable way. Today, someone goes out to the field, sticks a tube in the ground, takes a core sample, sends it to a lab and six weeks later you get a result.

“It’s sort of like the cutting edge of R&D right now – how do we measure these soil nutrient qualities in real time?” he said. “It’s really hard, no one’s cracked it. And there’s a lot of people working on it.”

Key AI enablers still to come

While some have criticized John Deere’s AI efforts, questioning whether its AI-powered machinery is too expensive or too complex to use, who owns the data gathered and whether workers will be replaced, Sanchez said that the reality is that finding good, dependable, skilled labor is one of the biggest challenges facing farmers today. Employment of agriculture workers, he added, is projected to only grow 1% from 2019-2029, slower than average for all occupations, while work on the farm can be very demanding during critical times of the year, requiring labor for up to 18 hours a day.

“Deere’s autonomous tractor and other advanced technology provides farmers with the flexibility to manage pressing tasks within their operation at those critical times, because the tractor can handle some of the work that they don’t have time for, or the labor to do, while they focus on jobs that still need their attention,” he said. “Farmers own their data and control who they share it with and when.”

In any case, Sanchez maintains that John Deere is still only “in the second or third inning” of implementing and commercializing AI-driven solutions.

“Right now in the market we have three or four meaningful solutions that have what you could really truly call powered by AI with all of the sensing technology, delivering significant value for hundreds and thousands of customers,” he said. “But  I think there are dozens and dozens more that are opportunities.”

He added that what’s “fun to think about” is that two of the limiting factors to scaling AI are having reliable training data sets and having readily available computing power. The more cameras and the more sensors you have because you have AI solutions, the more data you’re collecting. “So it’s sort of a network effect where the more you grow, the more opportunity there is with your dataset,” he explained.

Whether it is 5G or the next level of connectivity, Sanchez added that latency levels may finally allow John Deere to leverage the power of cloud computing in a way that’s truly real-time – “which for us is less than half a second,” he said, adding that would take the company’s AI efforts to yet another level.

“So, not only are we at the beginning of this, but there are a couple of key, massive enablers here that I think could potentially make this a lot more exciting,” he said.