Audiofunctions+: exploring math functions through sonification

Pedro A. García-Sánchez, joint work with Mattia Ducci, Ondřej Nečas, and Lukas Schölch
Departamento de Álgebra and IMAG, University of Granada, E-18071, Spain

Audiofunctions+ is a tool that facilitates the learning and teaching of mathematical functions for visually impaired students through sonification. It is one of the deliverables of the Erasmus+ Key Action 2 project SONAIRGRAPH, coordinated by the University of Turin, with participation from the Karlsruhe Institute of Technology (KIT), the GIS Community Association of Timisoara, and the Universities of Masaryk, Granada, and Valle d’Aosta.

Audiofunctions was created in 2014 at the University of Milan as a native iOS app to allow the exploration of functions on touch screens. The Polin laboratory of the University of Turin and the EveryWare Lab at the University of Milano, based on their research on the educational needs of students with visual disabilities in schools, extended Audiofunctions to Audiofunctions.web as a web application to be used in an educational context. This new version was made available at https://ewserver.di.unimi.it/audiofunctions . Further research, interviews, and user feedback, gave rise to Audiofunctions.web 2.0, which is accessible at https://audiofunctions.netlify.app . The first beta version of Audiofunctions+ is available at https://audiofunctions-plus.netlify.app .

As its predecessor, Audiofunctions+ is an open source and freely available web application that is meant for teaching mathematical functions in secondary school. Research will be conducted on both the usability and methodological aspects of this tool, in order to determine the impact in teaching and learning mathematical concepts related to the study of functions for visually impaired students. Audiofunctions+ can be used in any computer via a browser. The teacher can prepare working sessions for their students, and can be shared via link (or a json file).

Functions are inserted in text fields via an algebraic expression. Piecewise functions are described via algebraic expressions and inequalities where these expressions apply. The user can change the default view (the box in which the function will be represented) by using the WASD-keys for pan, and Z or shift-Z for zoom.The app plays a sound depending on the position of the cursor and the value of the function at the x-coordinate of the cursor.

If f is the function to be represented, then the pitch of the sound depends on f(x), the value of f at the x coordinate of the cursor. The x coordinate itself influences the placement of the sound in a stereo panorama (the sound will be reproduced on the left ear if we are at the leftmost position of the screen and will increasingly move to the right ear as we move to the rightmost position in the screen). If the exploration is made via the mouse, the volume of the sound depends on the distance between y and f(x), where y is the y-coordinate of the mouse cursor. Keyboard exploration can be either stepwise or smoothly with the arrow keys. The user can play the whole representation of the function in the current view box by pressing the key “b”.

The app offers two kinds of sonification: continuous and discrete. While the continuous sonification produces a sound constantly, the discrete sonification will only play a sound if the y-value of the function varies significantly to the last sonified position. A guitar sound is used for discrete sonification, while a clarinet is used by default for continuous sonification.

The app offers several earcons to help the navigation. When the user reaches the end of the view box (either at the right or left), a sound is played. Another sound is produced if, for a given x-coordinate, (x, f(x)) is not in the current view box. When the user crosses the y-axis with the cursor, another earcon is produced. A gentle noise is added to the sound when the value of the function is negative.

To ensure that important function points, such as isolated points, are not accidentally missed during sonification, there is the “Point of Interest” feature. This ensures that the corresponding point cannot simply be skipped over with the cursor. When a point is skipped during step-by-step navigation, a notification earcon is played to alert the user that a “Point of Interest” has been missed. In smooth navigation, gentle snapping has been implemented for these points.

The “Points of Interest” are recognized and set automatically by Audiofunctions+. Currently, these are limited to isolated points and undefined points of a function. In principle, however, teachers can define any desired points of interest in the JSON file.
With this functionality, an auditory equivalent to visual representations of specific points is to be achieved. This is because visual representations are usually clearly recognizable with a big dot, even though they represent only a very specific point in the function. The snapping plays that functionality for the sonification.

To enable fast access to all functionality via the keyboard, a CommandPalette and shortcuts were implemented. The CommandPalette can be opened at any time, using the shortcut ctrl+k (cmd+k on Mac) to search for actions. A fuzzy search with alternative keywords is used so that users do not need to memorize exact action names. Additionally, all actions can be navigated like a list with sublists within the CommandPalette.

Associated to constraints in a piecewise function, there is a set of “endpoints”, which are the points in the graph of the function determined by the ends of the intervals associated with the constraints. For instance, the function f(x)=x if x<1 and f(x)=2 if x>=1, produces two endpoints, one hollow and the other one filled out:

Functions and endpoints are drawn using JSXGraph. Sonification is performed with tone.js . The math parsing of the algebraic expressions and constraints for piecewise functions is carried out with the help of math.js . JessieCode is used to evaluate expressions. Pieceswise functions are translated recursively to javascript conditional expressions “C ? A : B” and then passed to JessieCode.

The user can have several functions defined in one session, but only the active one will be displayed and sonified. There are keyboard shortcuts to switch between functions, so that the student can compare their properties easily.

The authors are supported by the SONification for Accessible and Inclusive Representation of GRAPHs in Education (SONAIRGRAPH) project, which is an Erasmus+ key action 2 (project number 2024-1-IT02-KA220-HED-000244481), funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Education and Culture Executive Agency (EACEA). Neither the European Union nor EACEA can be held responsible for them.