Define a before and after vision of a business process, achievable only through ML/AI.

Each grid of storyboard includes a setup, transformation, resolution, and result. Data are clearly identified, though perhaps not yet explained, in the setup step. Each result is a logical conclusion, given the transformation and resolution provided.

Synthesize, expand on, and recreate the AI/ML implementations of others by drawing analogies to use cases in other domains.

Articles/papers linked to provide intuitive connections to the storyboards. The needs for ML/AI and “shift” in paradigm are cleanly articulated. At least one risk is provided for each use case.

Articulate how different ML/AI capabilities may be applied in a variety of settings.

Use cases provided show depth of thought into ML/AI’s differentiation. There is breadth in application (e.g., optimization vs. prediction, or NLP vs. computer vision) of various techniques.

Determine the potential of using ML/AI approaches by analyzing relevant data characteristics.

Justifications for data rankings in the Five V questions are well reasoned and explained.

Determine the types of AI/ML capabilities needed for a given problem, based on available data, and not vice-versa.

Approaches (e.g. classification, regression) are correctly selected based on the storyboards provided. Explanations correctly incorporate key terms such as “class”, “continuous”, “binary”, “objective”, “policy”, “state”, “reward.” Applications are appropriate and not forced.

Articulate sound judgments around feasibility vs. impact based on the data characteristics of a given use case.

“Grid” reflects discretion within the use cases provided and shows understanding of relative strengths. Strengths along each axis roughly map to ratings provided in prior exercise and are appropriate for the use case context.

Correctly identify the ML/AI capabilities needed to deliver a contemplated use case.

Machine learning capabilities align with the apparent objective. Non-ML capabilities are likewise used when appropriate. Annotations are used where connection isn’t apparent.

Show how data inputs and capabilities flow as inputs and outputs in a basic and intuitive fashion that is digestible to both business and technical stakeholders.

All capabilities have some form of input/output visualization. Different layers are correctly called out. Training data ties back to Project Step 2 discussion.

Demonstrate how technology and data flows facilitate the user experience by connecting “capabilities” back to actual “experiences” and correctly identifying “end users” as opposed to “stakeholders.”

The submission identifies the end user. The user experience layer is present. Output flows suggest how a human will be involved in the process and make (or perhaps simply observe) a final decision.

Assess trade-offs and prioritize or reprioritize a series of ML/AI use cases accordingly.

Student has completed all requirements for this step as directed. Final grid/board reflects trade-off assessments accurately.

Assess model effectiveness in machine-learning use cases. Develop longitudinal plans to measure the success of models.

Model effectiveness is clearly defined relative to some form of human or current-state benchmark. Model effectiveness and probability/confidence are measured separately, and not conflated. Model effectiveness is treated as an ongoing measure, and the notion of drift or loss of effectiveness is accounted for.

Explain model bias and overfitting as these concepts pertain to real-world scenarios . Suggest ways to prevent these scenarios on an ongoing basis.

Student’s explanation clearly demonstrates an understanding of the difference between real-world bias vs. model bias. Student’s explanation discusses both bias and overfitting, and highlights risks of either particular to this scenario. Bias and overfitting are both discussed as ongoing, longitudinal concerns. A monitoring/mitigation plan is present.

Address ethical concerns of AI/ML models in their work. Present approaches to mitigate these risks over time.

Ethical concerns are assessed as often related to, but not the same as, model effectiveness, bias, or overfitting concerns. Student clearly highlights who potentially impacted parties may be, and highlights all stakeholders — users, users’ customers, their own employees, etc. — who may be affected. Ethics is addressed as an ongoing, longitudinal concern and a process for monitoring and mitigation over time is presented.

Identify and solicit feedback from relevant stakeholders who may have little experience with machine learning or AI, but may be in a better position to assess the business impact of a use case.

At least five filled-out Google forms are submitted, one for each of the email-based, phone, or in-person interviews. Surveys, interviews, etc. cover at least three different use cases being discussed.

Synthesize and analyze feedback that may come from a less informed point of view regarding AI/ML.

Some form of graphical or otherwise intuitive representation of results is provided. Representation provides some form of indicative trend or conclusion that is apparent to the reviewer. Student draws clear, data-driven conclusions that are consistent with the analyses presented.

Prioritize use cases, based on external feedback.

Student emphasizes (implicitly or directly) the differences between feasibility and impact. Student appropriately “weights” feedback based on interviewees’ experiences.

Tell a cohesive narrative around how ML/AI affects a business, balancing business impact and feasibility.

Student demonstrates holistic knowledge of ML/AI throughout presentation. Wording and storyline of presentation is clear to reviewer. “Big takeaways” that go beyond one exercise or another are present.

Demonstrate full comprehension of prior exercises and why they can be helpful in evaluating an ML/AI business strategy.

Student should draw upon earlier work and learnings, possibly including prioritization grid, architectures, survey results, or any other steps. The use of these exercises should demonstrate a deeper understanding of their purpose in determining a cohesive strategy. Presentation should highlight (implicitly perhaps) how exercises helped to create the priority among use cases.