Handling user-generated content with Multer and AWS
So for the past few weeks I have been working on a project with a team of seven devs. The app we are building is educational platform for children to collaborate on a shared story. Each week the user is given reading assignment and is asked to contribute a piece of writing and an illustration based on a given prompt.
Up to this point I have been the sole back-end developer, so I was tasked with creating the necessary database, endpoints, and functionality for the API. I was modeling the database when a thought occurred to me: if this app went to production and had say 5000 users, there would be at least 20,000 photos being uploaded weekly. In the past when I was handling user content it had been on small personal projects not meant for production. The way that I handled user-submitted files on most of those builds was to just store them directly in the database as BLOBS (short for large objects) which as the name would suggest are not very space-efficient. On one past project I took a slightly different approach and used a library called multer to store incoming in a folder on the repo itself. This method was easier to work with but just as memory intensive as using the BLOB approach. I thought about those 20,000 incoming photos and realized that the ways that I had handled this problem previously weren’t going to cut it at scale.
I started looking into scalable approaches to user generated content and found a very handy library called multer-s3. Multer-s3 works similarly to multer but allows you to send files to an AWS bucket to be stored, that way the load of user generated data can be put on Amazon’s servers and not on your puny Heroku-deployed site. We were lucky enough to be provided with AWS credentials for this project so we could host our front-end on elastic-beanstalk so I figured I could use those credentials to implement a proper solution for our UGC.
First I had set up the S3 bucket for the files to be sent to. AWS is a massive platform with an absurd amount of features and I had absolutely no experience working with it. Navigating the management console was quite overwhelming at first. Thankfully with a widely used platform like AWS, there are always plenty of helpful guides online. So with the help of a tutorial I generated my access keys, created my bucket, and wrote out my bucket-policy and CORS-configuration.
Next I set up the multer-s3 functionality. Multer works by allowing you to accept form-data as a request body instead of JSON. It reads through the sent files, verifies the file-type, and then stores them at the desired path on your backend. Multer-s3 adds an extra step using the AWS SDK. It instead sends the files to an S3 bucket and returns the urls of the uploaded files. I entered all the AWS config file and built a test endpoint to receive some files.
I was ready to put all the pieces together but unfortunately I just couldn’t seem to get it to work. I fixed a few typos in my postman request and got it to where multer was processing the files but AWS was rejecting them. I read through probably 25 threads on stack overflow and github trying a bunch of fixes. After banging my head against the wall for hours I decided to put this feature on hold and I moved on to other things. For the three days or so, I would jump back in and try again for a few hours.
I tried so many different implementations. I built and rebuilt everything from scratch several times. I just kept getting stuck in the same place. My S3 bucket just didn’t want to accept my files. It was two weeks into the project and I had everything else ready to go for our first release canvas. I was ready to give up and build a non scalable solution but I decided to try and get it working one last time. I started over and carefully built everything out. I hit the same error and I threw some random, desperate console logs into my code. The source of my problems finally revealed itself and my palm slapped my forehead. My environment variables were undefined. I installed dotenv and everything started working perfectly.
I probably sunk way too much time into this and should have given up many hours before I did, but it felt great when everything finally snapped together.