Ultimate 7 Step AI Project Management Guide

Requirements gathering

The first step of an AI project is requirements gathering and documentation. As with any development project, requirements will set the foundation for the project and ensure it is able to be built to the stakeholders’ and users’ expectations. As part of the requirements gathering process, project managers should define:

• Project overview: Outline the purpose of the project, the desired outcome, and any specific goals and objectives to be achieved.
• Roles & responsibilities: List all project team members, including who the project stakeholders are, who will provide input for defining the project’s requirements, and who the levels of management are. Use task management tools to assign work to individual team members throughout the project..
• Constraints: Document any project constraints that could affect the project planning. This could include time, budget, or technology constraints.
• Technology stack: If you already have a tech stack selected, it’s important to document it at this stage. However, we recommend that the exact AI tools you use be left up to the technical team.
• Functional requirements: Thoroughly document all requirements that define how the system should function from a user’s perspective.
• Non-functional requirements: Include requirements that define how a system should perform in terms of its attributes, such as reliability, scalability, and security.
• Testing and validation plan: Explain the process of testing the software requirements, including the use of test cases and specifications.

Solution planning

Now that you have your project requirements defined, you should hand them over to the development team who will begin their own evaluation and solution planning work. Solution planning should roughly follow these steps:

• Conduct a feasibility analysis: Before proceeding with the development of the solution plan, it is essential to conduct a feasibility analysis to assess the technical, economic, and operational feasibility of the project. This involves the technical team evaluating the project's requirements, constraints, and risks. Based on the findings, the project scope and requirements may need to be adjusted.
• Identify the AI approach: Based on the feasibility analysis, the team should identify the most suitable AI approach for achieving the goals of the project. This may involve choosing between supervised, unsupervised, or reinforcement learning, or a combination of these approaches.
• Select the technology stack: Considering the constraints and requirements of the project, along with the AI approach that has been selected, the team should select an appropriate technology stack, which could include: one or more AI models, cloud platforms, data exploration tools, and development platforms.
• Design the architecture: The team should design the AI architecture that will be used for the project. This includes identifying the algorithms, models, and frameworks that will be used, as well as the infrastructure and tools required. The architecture should take into account the tech stack as well as the project’s constraints.
• Develop the solution plan: Based on the AI approach and architecture, the team should develop a solution plan that outlines the project's implementation details. This should include a detailed description of the AI approach, technology stack, development requirements, and evaluation strategies. It should be written in a way that can feed into the development of a data plan.

Data planning

With a solution plan in place, it’s now time to further develop the data plan for your AI project. Depending on the expertise of your team and resource management strategy, this work could be done by the same team that developed the solution plan, or the solution plan could be handed off to an independent data team.

Regardless, your data plan will define both the scale of data that is needed to achieve the level of accuracy required as well as the source of the data.

Think about where you will source your data. Will it require purchasing a license, data scraping publicly available websites, or will all assets be gathered from internal sources? There are many options to consider, including:

• Past data: Often AI capabilities will be built on top of an already existing product, which means you may have data that’s been collected by your product that you wish to use. All you would need to do in this case is organize and make it machine learning friendly (data cleansing).
• Manual data collection: This involves hiring data labelers and having them manually label datasets. This is often a costly process and can take a long time.
• Automated data collection: This involves using automated tools and scripts to scrape data from websites and other sources. This can be less expensive than manual data collection, but the quality of the data may not be as high.
• Purchased datasets: Buying datasets from third-party data providers can be a cost-effective way to get large amounts of data quickly, but it may require additional work to clean the data of any irrelevant or incorrect information.
• Generative models: Generative models are a type of AI algorithm that can learn the underlying structure of a given dataset and can generate new data based off of what it learns. The generated data can then be used to augment the original dataset for the AI project.

The data plan should also include details about how you will preprocess the data that you collect. The type of preprocessing required will depend on the type of data as well as your end solution. Generally, it involves cleaning and transforming raw data into a format suitable for use in an AI solution in order to reduce noise and improve model accuracy.

Implementation

Implementation is the process of translating the solution and data plans into reality.

Teams who are in the implementation phase of an AI project should follow these key steps:

• Gather data: Collect the necessary data for the project from the data sources identified in the data plan. Ensure you have enough data to achieve the level of accuracy you are expecting.
• Clean the data: Remove any irrelevant or redundant information from the data so it is suitable for use in the project. This is a critical step to ensure that the model is able to achieve the desired results.
• Train the model: If you’re building your own AI mode, this is when you will use the data collected to train the AI model so it can recognize patterns and make predictions.
• Develop the models: This includes selecting and tuning the parameters and hyperparameters to ensure the models are running as expected and are producing the desired results.
• Validate the models: You will do further testing and evaluation work in the next step, but iterative validation should be an active part of the implementation work. Validate the models by testing them on real-world data to ensure they are accurate and reliable. Make modifications in real time to enhance the model’s performance.
• Integrate the components: This includes ensuring that all the components of the project are properly connected, including the data pipelines.

Testing and evaluation

After the implementation phase, thorough testing should be done to ensure that the models are running efficiently, meeting the expected accuracy levels, and running within the expected time limits. Traditional black box and white box testing can be employed to test AI systems. This could look like utilizing one or more of the following specific testing methods:

• Model performance testing: Evaluate the model performance on test data/new data sets to measure parameters such as precision, recall, F-score, and confusion matrix (False and True positives, False and True negatives) against predetermined accuracy.
• Predictive model testing: Evaluate how well a predictive model generalizes to an independent data set. This is done by testing the model on unseen data, or data that the model has not seen during training. The goal of predictive model testing is to determine how accurately the model can predict the outcome of unseen data.
• Metamorphic testing: Test the system with mutated input data. The quality of the output of the system is then evaluated to see if it remains consistent with the expected output. The idea behind this type of testing is to identify any hidden flaws or errors in the system that might not be detected by traditional testing methods.
• Dual coding/algorithm ensemble: Use two or more algorithms to test the same data set. The algorithms are then compared and analyzed to determine which is most accurate and/or efficient. This type of testing is important to ensure the accuracy of an AI system and to identify any potential issues with the system.

One of the most challenging aspects of AI testing is obtaining test data sets that can be used for:

• Measuring accuracy and reliability. The data should be labeled by experts and tested against customer feedback in order to ensure accuracy.
• Identifying anomalies and potential issues with the AI system. It can also help to identify potential improvements or changes that need to be made in order to improve system performance.
• Keep performance in check. This is important in order to ensure the AI model is continually improving and delivering the desired results.

Deployment

Deploying an AI project is a vital step in AI project management. This is where you make your AI software available to your target audience, whether internal users or customers. The exact process for deploying an AI software tool depends on the type of AI tool, the environment in which it will be deployed, and the infrastructure required for the deployment, however, in all cases the following considerations should be weighted by the project leaders as part of the deployment process:

• Security: Ensure that the data is secure, the system is not vulnerable to attack, and that the users’ privacy is respected.
• Performance: Ensure the AI tool continues to perform as expected and meet the desired performance goals.
• Scalability: Consider the number of users, the amount of data, and the number of models that will be running simultaneously.
• Usability: Ensure the AI tool is easy to use and understand for the target audience.

Monitoring and maintenance

Like with any software, AI systems require ongoing monitoring and maintenance to ensure that they continue to deliver the desired value to users. The project manager should use the initial project deliverables, objectives and KPIs to measure long-term performance. We recommend that the product team regularly evaluate:

• Accuracy of the models: Compare the accuracy of the models against project benchmarks. Expect accuracy levels to change over time as the model is tested against more real data in a production environment.
• Efficiency of the data pipelines: Monitor the data collection process, the data transformation process, and the model training process to identify any areas of inefficiency.
• Performance level: Monitor the latency, throughput, and resource utilization.
• Security: Monitor the system for any potential security vulnerabilities, ensure that any data that is being processed is encrypted, and that any data that is being stored is secure.
• Changing requirements: Make any adjustments to the AI system based on newly available data, changes in business needs, or advancements in the industry.

Findings during ongoing maintenance could trigger the need for retraining the AI system with new data, updating the algorithms or updating the existing AI solution to deliver on new business requirements.