AI Enhances Shiny App Accessibility with Dynamic Alt Text

AI-Driven Alt Text Makes Shiny Apps More Accessible

A new approach using the {ellmer} package in R demonstrates how artificial intelligence can bridge long-standing accessibility gaps in interactive data applications. By employing large language models (LLMs) to generate tailored image descriptions—or alt text—developers can now make Shiny apps more usable for people with visual impairments, without increasing their maintenance burden.

The Role of Alt Text in Accessibility

Alt text, or alternative text, provides brief written descriptions of images, allowing screen readers to convey chart content to users who cannot see them. Effective alt text is concise, delivers relevant context, and avoids redundant phrases. While well-established in static web design, alt text for rapidly changing or interactive visualizations, typical in dashboards and applications, remains a challenge.

The Accessibility Challenge of Interactive Dashboards

Interactive dashboards and apps encourage users to engage with data in real-time, filtering, zooming and updating visualizations on the fly. However, this very flexibility often complicates accessibility—particularly for dynamic graphics, where hard-coding descriptive alt text for every possible visualization is unfeasible.

Integrating {ellmer} and LLMs for Automated Alt Text

The {ellmer} R package, developed by Posit, enables seamless access to cloud-based LLMs from within R. The article demonstrates how {ellmer} can be coupled with Shiny—a popular R web application framework—to automatically generate alt text based on the current state of a ggplot2 visualization.

The recommended workflow involves:

• Setting up an LLM connection (the author used Google Gemini's free tier, but other models are supported).
• Creating a function that:
  • Renders the relevant plot at a consistent size and quality.
  • Passes the image to the LLM using a prompt to generate a summary.
  • Returns a simple fallback in case the API call fails.
• Connecting the resulting alt text to the plot output, ensuring users interact with updated, context-aware image descriptions in real time.

The article provides code snippets illustrating this process. Error handling is integrated, ensuring the description is available even if the LLM is unreachable. For high-traffic or complex apps, further recommendations include caching results, enhancing prompts with variable names and units, and allowing for manual overrides on critical visuals.

LLM-Generated Alt Text: Quality in Practice

Through practical examples, the author demonstrates that LLMs are capable of generating insightful, accurate alt text. For instance, a simple scatter plot of iris data yields a description identifying axis ranges, clustering patterns, and category distinctions—all elements that offer meaningful context to screen reader users.

However, the author stresses that AI-generated alt text should complement—not replace—human oversight. Letting end-users know that the descriptions are machine-generated encourages critical engagement and helps maintain trust.

European Context and Broader Impact

As digital services become ever-more pervasive across Europe, meeting stringent accessibility standards—such as those required by the European Accessibility Act—is critical. Tools that automate and streamline accessible design, like {ellmer} and LLMs, not only improve inclusion for individuals with disabilities but also support organizations in meeting legal requirements and best practices.

Conclusion

The integration of LLMs for dynamic alt text is a small, practical step that can have outsized impact on inclusion in data-driven applications. By leveraging AI tools thoughtfully, developers can create interactive experiences that remain accessible, robust, and compliant by default.

For the original article and in-depth examples, visit The Jumping Rivers Blog.