Session 1

• Create a musical instrument and come ready to perform it for 30s – 1min.

What I Did

I wanted to create a musical instrument based off of the gesture of writing emails. It’s one of my least favorite tasks and I realized the small joys I would get from it were in the sounds my keyboard or mouse would make with the repetitive gestures of my typing or scrolling through my inbox and clicking and unclicking to (mostly) delete emails.

I made a Google Chrome Extension that triggers sound based off of keyboard and mouse interactions I programmed to match these gestures I associated with emailing on my computer.

[Loom video recording of person typing an email as sounds are created.

](https://www.loom.com/share/277ef76e8e4f4468a7a9d5284a8c148d?sid=dc621f58-a780-4026-ac6a-cde6779b43be)

Loom video recording of person typing an email as sounds are created.

Screenshot of Chrome extension running locally with options to see Details, Remove, or Errors and also to toggle extension on or off.

I’m currently using a p5.js sketch to test out the code and then copying over the sketch.js code and running it locally in a directory set up to be uploaded as a chrome extension.

What Worked

• Getting inspired

  • Sound examples from my ICM 2023 Media section taught by Ellen Nickles
  • This is where I found the following examples that I was influenced by:

    • http://jazz.computer/

    • https://typatone.com/

      

      Interface for Typatone, a website where you can type and music is created through tones.

    • https://patatap.com/

  • I was also inspired by Herdimas Anggara’s guest speaker presentation/performance in Applications last semester
  • Class readings
    • Music as mirror of mind, by Laurie Spiegel
    • Theremin in the Press, by Jaime Oliver

• Referencing library docs and past

  • Chrome Developer Google Chrome Extensions Docs

    

    Screenshot of VSCode with manifest.json file in the extensions directory.

  • Coding Train Google Chrome Extension video

  • Codecademy: p5.js creating a local project

  • Tone.js Library

  • p5.js reference (specifically Events)

  • ICM Sketch of mine from last semester

• Breaking the project down

  

  Screenshot of p5.js editor displaying sketch.js file with functions for handling key press, mouse clicks and wheel events and triggering sounds as well as console log messages.

  

  Screenshot of console from the p5.js editor showing the data from WheelEvent triggered by the mouse wheel moving.

• Testing along the way

  • Testing as I went along helped me debug errors when uploading the code I had working in the p5.js editor and locally on a web page after I uploaded as a local extension in developer mode.

• Once I had the browser extension set up I was able to adjust the code one feature at a time to figure out which sounds felt right with the intention for the instrument. Some adjustments I made at this step were:

  • 💡 Updating the mouse to trigger different notes on click and release versus using mouse clicked which only fired once after an entire mouse click event. Doing this gave me more flexibility in using the mouse to create a melody and I enhanced this by making the events trigger two different notes.
    • Currently the release trigger plays one whole note higher than the press which to me feels both like a weight lifted but also always unfinished (this is similar to how I feel when finishing writing an email 😅)
  • 💡 Updating the mouse wheel to play the same note but in different octaves. To me this matched the playfulness I feel when scrolling and matched the movement I feel of this gesture contrasted with typing or clicking the mouse.
  • 💡 Pressing the Return or Enter key updates the note that is played when typing which to me matched the change a line break adds.

For some final testing before class I sent myself an email and replied to it. I would love in the future to create these types of “email poems” while typing with the instrument enabled in my browser. Below is the text and screenshot from the first one, and next time will remember to record the audio at the same time so I can compare them.