Artificial Intelligence, Lean Start-up Method, and Product Innovation – Business & Generative AI

https://www.youtube.com/watch?v=eUWEuU8p6YI

Lean Startup vs AI for Innovations

All right. Hello, everyone. I am the final person before the reception, so I will try to be fast. So today I want to talk about a paper that I co-authored with Professor Lynn Wu. The title is Artificial Intelligence, Lean Startup Method and Product Innovations. It has little to do with generative AI, but we would like to present this to provide kind of a thinking framework or structure to think about how artificial intelligence may help with innovations, OK? Is this working? Yes. So product innovation is key to success in most companies, especially startup companies. So traditionally, many startups companies are using the so-called Lean Startup Method to help with product innovations. So it was introduced by Eric Rice in his famous book, The Lean Startup. And this method actually advocates product innovation through market feedback and continuous experimentation, instead of relying solely on the entrepreneur’s personal insights or a rigorous business plan.

So there has been some commonly adopted practices for the Lean Startup Method, for example, prototyping. I guess many of you have seen or have been using this kind of beta version of software.

Another widely adopted method in the Lean Startup Method is A/B testing. A lot of companies are running A/B testing extensively to continuously improve their products. So there has been many papers documenting that Lean Startup Method can be helpful for product innovation by alleviating the market uncertainty or to help improve the product qualities.

However, there has also been concerns about the caveats of Lean Startup Method. For example, some people argue that the Lean Startup Method is really lack of market insights, and also it can be time consuming to use the Lean Startup Method to finally converge to an optimal market niche. And also, some other people argue that the Lean Startup Method is associated with repetitive labor work, and it restricts fast iteration in product development. So you can think of Lean Startup Method as the gradient descent in product innovation.

So basically, you start at some random initial value place, and through continuous experimentation or prototyping, you receive the market feedback, you do the gradient descent, and finally, you might end up at the local optimal point.

AI in Product Innovation: Exploration vs Exploitation

So as I just mentioned, there are also a lot of caveats of the Lean Startup Method. So besides the Lean Startup Method, many companies are also using AI to help with their product innovation. So in this paper, we define AI as the advanced computing power enabled by machine learning algorithms that can help companies take predictive actions. And this AI capability is associated with high prediction accuracy but low interpretability, just the same as the large language models that we have observed. It’s usually associated with a pretty high prediction accuracy, but it’s usually very hard to interpret what is happening inside of the model. So some companies are using AI to help them explore external market opportunities. For example, Genki Forest is a beverage startup in China. The company has been using AI to analyze social media data in China. And they find out that on social media platforms, when young people are talking about beverages, they are also very likely to talk about sugar, diabetes, and obesity, these kind of keywords. So their later creation of zero-sugar sparkling water became a huge market success.

In another example, Airbnb has also been using AI to help automate and optimize their existing processes. For example, they have been using AI to automatically classify room pictures. For example, a room picture like this would be classified as a living room. Another room picture would be classified as a kitchen. So using this AI algorithm, Airbnb can provide a much better search experience for customers on their platform. And this has become a key function in their premium version of product, Airbnb Plus.

So based on these two examples, you can see that some companies are actually using AI for explorations. Basically, they’re using AI to search and discover unknown new insights from the market. While some other companies are using AI for exploitations, basically, they’re using AI to automate and optimize known existing processes.

AI for product innovation: benefits and substitution

So we have seen that some examples show that artificial intelligence can help with product innovation. But based on what we have already observed, and also the presentation yesterday, Eric Nelson has introduced, we know that artificial intelligence is helpful when the data is reliable. But in the novel areas, usually the data is sparse and unreliable. And this is also what we observe in the industry. The majority of firms do not actually observe benefits from their AI investment. So in this paper, our research question is basically twofold. On the one hand, we would like to ask whether AI can really help with product innovations. And moreover to that, we also want to ask, does AI substitute or complement the traditional lean startup method? Because the answer to this question can help us to explain why some companies are benefiting more from using AI while others do not.

So let me give you an overview about this paper. So we have curated information for about 2,000 startups in China from 2011 to 2020. So for the deepened and variable, we measure the product innovations for the startups using their software copyrights for software companies and also product trademarks for companies developing physical products. So there might be concerns that those startups may just develop a lot of useless copyrights or trademarks into the market. So we also validate our results using their overall productivity level, including their funding amount and also their web traffic from the major search engine in China.

Measure AI capability and lean startup practices

For the independent variable, we measure the AI capability and also their lean startup method adoption level for each company in each year based on their job postings, social media articles and news reports for those companies.

So for the AI capability, there has been a stream of literature documenting the AI keywords that you can use to refer to their AI capability.

For the lean startup method level, we have categorized two main practices for the lean startup method used by most companies. One is prototyping, basically building trial products to test the market feedback. The other is controlled experimentation, which is basically continuous A/B testing to receive the market feedback to iteratively improve existing products.

AI policies drive China’s AI adoption

Our main major identification strategy is the AI policies issued by local governments companies in China. So in the past decade, there has been a huge shift in the AI adoption in many local business in China, which has been driven mainly by the local policies in China that has been encouraging AI adoptions. An example policy would be R&D expenses in core tech areas, such as artificial intelligence, can be deducted as 175% in tax reporting. So such policies help to encourage AI adoption by lowering the adoption cost of artificial intelligence capabilities in the local business. And we can also observe that such policies are also not highly correlated with the economic performances for the provinces in China.

So let me give you an overview of the results.

So we have divided our sample company into companies who have not adopted lean startup method versus companies who have adopted some level of lean startup method. And we find out that after their adoption of artificial intelligence, the performance for product innovation for software startups has significantly improved if they have also adopted lean startup method. But such results is not significant if the company has not adopted lean startup method. And such results are also consistent for companies developing physical products. So such results suggest that artificial intelligence and the lean startup method might be complements to each other to help with product innovations. And furthermore, we have also verified this using regressions. So we have done this complementarity regression. And we find out that if a company has already adopted lean startup method, the company is more likely to adopt AI capability as well. And such results are not only consistent for software companies, but also for companies developing physical products. Moreover, if the company has adopted both artificial intelligence and lean startup method, their product innovation performance would be significantly higher. And such results are also consistent using instrumental variables. And we would like to highlight here that we find out that if you are only adopting artificial intelligence without lean startup method, the benefit can be limited. And such results are also consistent for companies developing physical products. So why is that? What is AI doing here? Why is AI helping with product innovation?

AI, Prototyping, and Product Innovation

So in order to examine the underlying mechanism, we have categorized two types of product developments inside those companies. So basically, for companies, usually they will develop a type of novel product and another type of incremental products. So you can think of novel products as the new product of the company that the company is entering a substantially different market niche. Well, for incremental products, they are updated versions of existing products where the external market demand has already been validated previously, and the company is still staying in the same market niche. For example, the development challenge for developing the first generation of iPhone can be substantially different from developing the 15th generation of iPhone. So the major challenge for developing a novel product would be the market uncertainty because the company really needs to find out a market niche with real market demand. Well, for developing incremental products where the market demand, market uncertainty is already validated, the major challenge for the company is to fast iterate their product to improve their product quality. And for the novel products, your customers are usually early adopters of the product who are usually passionate fans about your product and are willing to accept some quality caveats. But for incremental products, your customers are usually mainstream customers who usually have higher requirements for the product qualities.

For developing novel products, as we just mentioned, the major challenge would be the uncertain market demand. The exploration-oriented AI, as we just mentioned, can help to provide the market insights and guide the company where to use the Lean Startup method. As an example we have just mentioned, in the company of Genki Forest, the company has been using AI to analyze social media data and find out that young people care a lot about sugar, diabetes, diabetes when they are talking about beverages. Well, on the other hand, prototyping can also help to validate the AI-enabled search results and further identify optimal market niche. For example, in the company of Genki Forest, after they find out that young people care a lot about sugar, they initially launched a zero-sugar milk tea into the market. But the market feedback rejected their initial hypothesis, so that they have to pivot into some other directions. So after several rounds of trial and error, they finalized at zero-sugar sparkling water, which later became a huge market success.

On the other hand, for developing incremental products where the external market demand is already validated, the major challenge is to experiment through the universe of possible input features so that the company can find out the optimal subset of input features to create a higher quality product. In this case, the exploitation-oriented AI can help to automate the repetitive work and optimize existing development processes so that the company can expedite their iteration process. Well-controlled experimentation, such as A/B testing, can help the company explore the universe of input features and further improve their AI algorithms. For example, in the previous case of Airbnb, the company has been using AI to automatically classify room pictures. However, instead of using AI to go over every single pixel of these high-fidelity room pictures, Airbnb has also been running A/B testing extensively to run their AI to run their AI algorithm only on certain components of these room pictures so that they can make the prediction for the room classification in a much faster speed with high user satisfaction rate.

Alright, so basically we have decomposed the type of product innovation inside companies, so we also need to measure novel versus incremental products in our sample. So basically we use the version 1.0 software copyright for the novel product in the software companies, and we use the number of trademarks that contain a new subclass to measure the number of novel products in companies developing physical products. We have also decomposed exploration versus exploitation oriented AI. So an example of exploration oriented AI job would be a business intelligence or market analyst that conduct exploration market analysis with NLP technique. While an exploitation oriented AI would be a software engineer or AI engineer that are responsible for using machine learning to optimize R&D processes. For example, image denoising. We have also decomposed prototyping and controlled experimentation. So for prototyping, a typical job posting would be product engineers that are familiar with Google MVP platform and other platforms. So MVP here stands for minimum viable product. And controlled experimentation would be a data scientist that have an experimentation mindset and familiar with A/B testing.

We find out that for those job postings that require prototyping, they seldom also require skills related with controlled experimentation and vice versa.

Okay, so let me briefly show you some of the results. So we find out that prototyping complements with exploration oriented AI in developing novel products, but not controlled experimentation with exploitation oriented AI. And such results are also consistent for developing novel products. On the other hand, controlled experimentation complements with exploitation oriented AI for developing incremental products. And such results are also consistent for companies developing physical products. So in summary, we find out that exploration oriented AI complements the prototyping for developing novel products. Exploitation oriented AI complements with controlled experimentation for developing incremental products.

AI complements general lean startup.

Overall, overall, the general AI complements with a general lean startup method for product innovations. And we do not find cross complementarities in this system, probably because of the different context knowledge and skills required for different capability. And also the different R&D stages and objectives for different capabilities.

AI adoption analysis using IV and news

That’s all I have today. Thanks a lot for listening. I have a moderator mic so you can have questions. Any questions? Or we can go directly to the reception.

Oh, yes. Gavin, great work. I was wondering if there’s any, I mean, you probably have addressed, you mentioned that you did the instrumental variable analysis. So those who are self-selecting into AI and lean, there may be some intrinsic reasons why they are actually self-selecting into doing those.

Right, right. Yeah. Yes. So definitely that’s the case. So that’s why we try to use the instrumental variable approach. So one thing about using the AI policies is that policies are usually unpredictable for the local people. So that when there is a policy there, it can be considered as a shock, like external shock to the companies to adopt those kind of capabilities. So besides of using the AI policies, we have also used the news reports for your competitor company that has adopted AI as another instrumental variable. Because if your competitor is using AI, then this might be an incentive for you to also adopt artificial intelligence. So this is another instrumental variable we use for AI.

For the Lean Startup method, because of time I didn’t mention here, we actually use the Lean Startup method level for the companies who share the same investor with the focal company. And also this company should be in a different location and a different industry to avoid the systematic shock as AI has.

Yes. Thanks. Okay. Time for receptions. But thank you so much for coming. And it was a great, you know, I learned so much from these talks. I think the conclusive, conclusive remark is like AI really matters, but how it matters, it depends. Thank you. Thank you.