Team 6 Project 2
Using Big Data technologies, and Twitter's Trend API endpoint, project 2 consists of retrieving responses from Twitter with the Spark Structured Streaming API at 10 minute intervals, transforming the response with Spark by converting a single object with a location value and an array of 50 trends into 50 row entries and writing them to CSV format The output is then analyzed using Spark SQL to extract trend patterns to find the trends average rank, highest rank, hours trending, number of tweeets about the trend and top trends. The output of the project include Spark jarfiles, both pre and post processed datasets and documentation to repeatable and used on larger datasets and datasets that streamed on an ongoing basis. The questions examined follow: Can we identify patterns in trending topics being popular in one location and then moving to others? If so, can we identify separate regions for these patterns? (ex.: North American patterns, East Asian patterns, etc.) What about global scale patterns for specific events? ex.: Burning of Notre Dame-how did this topic travel according to twitter usage?

Roles / Responsibilities

• Utilized Postman and it's Collection Runner functionality to script the automated retrieval of Twitter Trend API events.
• Designed a Scala Spark Runner program to input raw JSON response files for processing.
• Extracted Twitter's JSON Trend Api response and fitted it to a Spark schema.
• Transformed the data from one object with a Location and fifty Trend topics to fifty objects with a Trend topic at that location.
• Exported data into the local file system as CSV.
• Destructured SparkSQL query function into separate Scala modules and packaged them as individual JAR files.
• Tested functionality of SparkSQL JAR files in Google DataProc.
• Produced and reported on analysis results.
• Implemented Scala, Spark on Windows, and Ubuntu Linux.
• Source control and versioning handled using Git and GitHub.

Environment / Technologies Scala, sbt, YARN, Parquet, Spark SQL, Spark Streaming, Spark, Google DataProc, Google Cloud Storage

---

• Github repository

Project involves a series of data analysis questions about the 2020 COVID pandemic. Analysis is performed using Spark applications (predominantly Spark SQL) on static and streaming data sources, making use of a Spark cluster and a Kafka cluster running on Amazon's cloud. Analysis questions and goals follow: Which Regions handled COVID-19 the best assuming our metrics are change in GDP by percentage and COVID-19 infection rate per capita. (Jan 1 2020 - Oct 31 2020)? Find the top 5 pairs of countries that share a land border and have the highest discrepancy in covid-19 infection rate per capita. Additionally find the top 5 landlocked countries that have the highest covid-19 infection rate per capita. Live update by Region of current relevant totals from COVID-19 data. Is the trend of the global COVID-19 discussion going up or down? Do spikes in infection rates of the 5-30 age range affect the volume of discussion? When was COVID-19 being discussed the most? What percentage of countries have an increasing COVID-19 Infection rate? What are the hashtags used to describe COVID-19 by Region (e.g. #covid, #COVID-19, #Coronavirus, #NovelCoronavirus)? Additionally what are the top 10 commonly used hashtags used alongside COVID hashtags? Is there a significant relationship between a Region’s cumulative GDP and Infection Rate per capita? What is the average amount of time it took for each region to reach its first peak in infection rate per capita?

Roles / Responsibilities

• Point person for Docker, Kubernetes, Amazon S3 and Elastic Stack (Beats, LogStash, ElasticSearch, Kibana).
• Designed a Scala Spark Runner program to input raw JSON response files for processing.
• Write and maintain documentation for Docker, Kubernetes, Running Spark jobs on Kubernetes (Amazon EKS). [github directory]
• Sourced documents, sample files, and wrote documentation for ELK, Spark, Git, DataProc, and Kubernetes/Docker.
• Spark-Submit jobs to Spark cluster running on AWS Elastic Kubernetes Service (EKS)
• Ran Spark jobs on Google DataProc using Google Cloud Storage.
• Utilized Amazon S3 to store Datasets and Spark JARs
• Ran Spark jobs on Amazon EMR in Cluster mode as well as Step Execution mode.
• Collaborated using Git on Github to manage source control between multiple teams.

Environment / Technologies Scala, sbt, Parquet, Spark SQL, Spark Streaming, Spark, AWS S3/EC2/EMR, Google Cloud Storage/DataProc, git, Kubernetes, Docker, Elastic Stack

---

• Github repository

A stock purchase app, built for Stage 2 of the NYC TTP application.

• Implemented Sessions and Bcrypt authentication
• Created RESTful API and data retrieval from IEX Cloud API for financial data
• Built with RoR server, Javascript/React front end, Axios Promise based HTTP requests
• Deployed to Heroku (server) and Netlify (frontend)

---

• Demo site
• Github repository
• Video walkthru at YouTube

A web app to design teams with Agile roles.

• Conceptualized app idea and MVP requirements
• Implemented a Ruby on Rails RESTful API server backend
• Developed a Javascript/React frontend Single Page App (SPA)
• Deployed to Heroku (frontend and server)

---

• Demo site
• Github repository (frontend)
• Github repository (server)
• Video walkthru at YouTube