Revisiting Mythical Mysfits

First Encounter

This is a story of how I encountered modern applications and how I started to learn about microservices. On my second year in Singapore Polytechnic, I took an AWS Cloud Foundations elective module. We were given an assignment to pick a hands on tutorial from AWS's catalogue.

I browsed around for a few days and stumbled upon an interesting one. It was called "Build a Modern Application on AWS" It has been removed from the catalogue, but the repository is still available on github.

What is this project?

If you arrived here from my resume, you probably already know the basic gist of this, but I will iterate it again.

Mythical Mysfits is a modern web application built using an API gateway microservices pattern deployed on AWS Cloud.

It was divided into 7 modules on the github repo, but only 5 were shown on the website and that was the extent of what I did.

Constraint

I was using an AWS Starter Account, which has limited services, but AWS provided us with free credits.

Implementation

This project uses Cloud9 as the IDE. Although I believe that we can also setup the project locally with the AWS SDK as well.

It is broken down into 5 steps:

1. Hosting the static website on S3
2. Creating a service with AWS Fargate + CI/CD Pipeline
3. Creating the database
4. Adding user authentication + hooking the microservices and AWS Cognito to API gateway
5. Capturing user clicks

Problems and Solutions

This article would be extremely long if I were to explain everything that I have done on this project, so here are some useful links to my project diary and pull requests. However, if you are interested in the explanation, I just drop me a text or email and I will try my best to write it! I have logged my progress, issues, and solutions on notion. I also created a pull request on the repo to help other students and teachers. Another user has also created a pull request implementing the tasks that I could not figure out at the time.

Improvements?

Here are some ways that I would like to improve the project if I could do it now:

1. Setup CloudFront to serve static website from S3
2. Followed module 6 & 7 to add more functionality
3. Modify the CloudFormation template to build the ECS cluster with EC2 with auto scaling
4. Use a free IDE to setup the project locally to save cost
5. Use github actions to automate the CI/CD pipeline because the free tier has 2000 CI/CD minutes/month

Intrigued by the concept of microservices and the start of my overengineered projects

I went crazy on this concept and tried many new things to learn as much as possible during my internship at AI Singapore. Attached above is an overengineered scraper that use RabbitMQ as its messsage broker, run with docker-compose, with data stored in both MySQL, and MongoDB. The services were written in Flask with 1 REST endpoint to start scraping and the rest are just the services communicating to each other through the AMQP protocol.

I bit off too much than I can chew and fell into LOTS of rabbit holes. I ended up extending this to export the docker logs to elasticsearch and display them on Kibana. It was honestly an overkill but I did not know that python's logging library could write into a file. I had a tunnel vision at the time because I wanted to learn as much as I can, but I am more open to looking into different options and using the easiest solution and improve on it now.

If I were to redo this, I would have written some simple python scripts (which I did, ...in the beginning) to scrape the data and store it in a database then move on to other tasks.

What I Have Learned Since

After talking to my mentor and friend, I have realized that not every application has to be built into microservices. I have learned that there is quite a bit of planning that goes into the design of a microservice, and it is difficult to maintain if the development team is too small (even more difficult when it is a one person team -- learned that the hard way 😅).

Demo

Video: Explanation + Demo