Improving the Duolingo Music Course Experience: A Practical Approach

 

Improving the Duolingo Music Course Experience: A Practical Approach

After three months of studying music with Duolingo without prior knowledge, I've noticed that the course provides valuable skills, such as associating notes on the staff with their names, keyboard positions, and characteristic sounds. However, I believe that the practical part, specifically playing songs, needs improvement.

Problems and Proposed Solutions:

1. Lack of access to sheet music at reduced speed: For effective practice, access to sheet music and the ability to play it at a slower speed is crucial. As a temporary solution, I have transcribed some songs and shared them on this blog for anyone who needs them.

2. Limitations of playing on the phone screen: The limitations of tapping on the phone screen are obvious. It's difficult, especially for those with larger fingers, and stunts musical growth. We are limited to only 200 Duolingo songs. To really progress and explore a broader repertoire (over a million songs), we need a full-size keyboard. I heard that Duolingo is developing MIDI support, which would be perfect.

3. 
   

Interim Solutions with MIDI:

While Duolingo implements the MIDI interface, I have explored alternative solutions to connect a piano keyboard to a mobile phone. Below, I describe two experimental systems:

2.a. System with Arduino Leonardo:

I have conducted tests with a toy keyboard and an Arduino Leonardo. This system simulates mouse movement on the phone screen. When a key is pressed on the keyboard, the Arduino sends a signal for the mouse cursor to move to the corresponding key on the phone screen and simulate a click. The click is held while the key is pressed.

Video Demonstration:

This system has the limitation of not being able to process multiple keys pressed simultaneously, which prevents the execution of chords.

This code is adapted to my specific configuration. To use it with other equipment, modifications will likely be required.

Adapting the Toy Keyboard:

For testing, I purchased a 37-key toy keyboard for 25 euros. Although the keys are slightly smaller than those of a standard keyboard (14 cm for seven white keys compared to 16.2 cm on a standard keyboard), they proved useful for this purpose.

Identifying the Switch Matrix:

It is crucial to know the arrangement of the keyboard's switch matrix (rows and columns) to correctly connect the Arduino. This information can be obtained by examining the printed circuit board (PCB).

Identification Process:

1. Photographs: Take photos of the PCB, focusing on the copper traces, switches, and diodes

2. 
   .
   
   
   

3. Connection Diagram: Using a drawing program, color each continuous trace with a different color, starting from the pins where the flat cable is connected. The vertical lines represent resistive bridges.

4. 
   [Colored image with the trace of each track in a different color]
   
   
   
   
   
5. Detail-pin-numbers, same number on PCB and I/O pins on Arduino Leonardo
6. 
   

3. Electrical Schematic: From the previous diagram, create a simplified electrical schematic showing the connections between the keys (rows and columns).

4. 
   
   
   

Connection with Arduino:

In my configuration, I use 13 pins of the Arduino Leonardo: pins {0,1,2,3,4,5,6,7} for the columns and pins {8,9,10,11,12} for the rows. It is essential that the pin order matches the arrangement on the keyboard's PCB and the Arduino pins. This leaves 7 I/O pins available for future improvements, such as a button to reset the mouse position or switches to select different Duolingo screens.

Verification of the Configuration:

To ensure the correct correspondence between the matrices numRows, numCols, keymap, rowPins, and colPins with the keyboard, an Arduino program can be used to display the number of each key pressed in the "Serial Monitor."

Source Code (Basic Version used to test my keyboard, shows the key number pressed):

Adapting to Screen Resolution:

The phone's screen resolution requires adjustments to the poslandx[] and poslandy[] matrices. This process is done practically:

1. Once the program identifies the keys and moves the mouse, the (x,y) coordinates where the mouse should be and where it actually is when a key is pressed are measured.

2. The x and y ratios are calculated.

3. The values of the poslandx[] and poslandy[] matrices are multiplied by the corresponding ratios.

4. The program is uploaded to the Arduino, and the process is repeated until the mouse is positioned correctly on each key. Although it may seem laborious, this process is streamlined with a spreadsheet.

Source Code (Basic Version, it reads the keyboard and moves the mouse pointer):

Link to access the code

Practical Tips:
1,- These tests are only possible with Arduinos with HID (Human Interface Device) capabilities that allow mouse control. The Arduino Leonardo has this capability.
2,- The program is designed to be used with the phone in the "landscape" position. The phone must be placed in this position from the moment it is turned on and remain so. After the song we want to play appears, indicating that we should press a particular note, we connect the USB cable from the Arduino Leonardo to the phone and observe how the mouse pointer moves until it is positioned in the upper left corner, which will be the origin. If it moves to the lower left corner, it will search for the origin in the "portrait" position and will not function correctly because the coordinates will not be valid.
3,- After some use, it may be necessary to return the mouse pointer to the origin. To do this, we open a song and in the waiting situation as described above, disconnect the USB cable from the Arduino to the phone, wait at least twenty seconds, and reconnect it. If done correctly, we will see the mouse pointer positioned in the upper left corner of the phone screen in the "landscape" position.
4,- It is recommended that the phone be fully charged, as the phone provides the power to operate the Arduino.

Mechanical Model (Under Development):

I am developing a simple mechanical model based on steel cables guided by hoses, similar to the bicycle brake system or remote camera shutters. The keys will be equipped with springs to return to their resting position. The actuator will be a flat-head drawing pin (with screen protection). To simulate human finger contact, the steel cables will be electrically connected to the phone's ground. This model is awaiting the delivery of the springs from the local Leroy Merlin store, to whom I paid in advance almost two months ago.