1 of 1

[1] Z. Lv, H. Song, P. Basanta-Val, A. Steed and M. Jo, "Next-Generation Big Data Analytics: State of the Art, Challenges, and Future Research Topics," in IEEE Transactions on Industrial Informatics, vol. 13, no. 4, pp. 1891-1899, Aug. 2017, doi: 10.1109/TII.2017.2650204.

[2] A. Kaplunovich and Y. Yesha, "Cloud big data decision support system for machine learning on AWS: Analytics of analytics," 2017 IEEE International Conference on Big Data (Big Data), 2017, pp. 3508-3516, doi: 10.1109/BigData.2017.8258340.

[3] Anastasiu, David C.; Christakopoulou, Evangelia; Smith, Shaden; Sharma, Mohit; Karypis, George. (2016). Big Data and Recommender Systems. Retrieved from the University of Minnesota Digital Conservancy, https://hdl.handle.net/11299/215998.

[4] Jon Handler, Introduction of Log Analytics, https://aws.amazon.com/blogs/database/author/handler/.

[5] Nick Carstensen, A log management and log cycle with AW, 2019.

CloudWatch monitors the AWS resources and applications that we run on AWS in real time. A thorough analysis of the log file is performed in S3 bucket with Query with S3 and with Athena query editor. These results show that the data is parsing successfully through the amazon kinesis firehose which is used as a delivery stream.

Overall, through this project, we are trying to highlight the role of big data services in various application scenarios and the advantages of real-time log streaming and processing provided by Amazon Kinesis. The integration with other AWS services like S3 and CloudWatch ensures seamless integration and eliminates the need for traditional infrastructure. We think that there is a need for further research to implement a big data-based recommender system for personalized recommendations and address data loss issues during data transport. Overall, the study demonstrates the significance of utilizing cutting-edge cloud services integrated with data for enhanced functionality and efficient management in web applications.

Real-time Big Data Log Analysis using AWS cloud

Project Advisor: Weider Yu

Introduction

Monitoring and application log analysis are essential for the reliable operation and scalability of web applications. By providing a comprehensive overview of application performance and facilitating timely identification of potential issues, monitoring and log analysis can help to improve the overall user experience.

This project focuses on exploring the key aspects of monitoring and application log analysis, with a specific emphasis on their application in building robust and targeted notification systems. We also investigate the role played by big data services in various application scenarios.

Methodology

Technologies used

Key References

Acknowledgements

On behalf of our team, we would like to extend our sincere gratitude to Professor Weider Yu for guidance throughout the project. His inputs and feedback have been pivotal in development of our project and guaranteeing the excellence of our efforts. We are truly grateful to work under his supervision. We also express gratitude to San Jose State University for giving us an opportunity and resources to work in a collaborative environment for the project.

The proposed application is a web-based application that enables users to search local restaurants, apartments etc. based on their preferred ethnicities. The application allows users to create a profile, signup to the application, login, and rate restaurants they like to allow other users to find recommendations.

The program is a cloud-based application that uses cloud services from Amazon Web Services (AWS). An EC2 instance with Kinesis built in is created whenever a user interacts with the web application or there is activity. Python scripts and dependencies are downloaded from Amazon S3 together with kinesis. IAM policies are adhered to while creating Kinesis data streams. This EC2 instance streams application logs, and Kinesis data analytics is built to do real-time website analytics.

The web application system is developed using ReactJS for the frontend and NodeJS for the backend.

Project Design

The frontend design for our web application is developed using ReactJS and Redux. React JS is a component-based library that makes it easy to create complex user interfaces, while Redux is a state management library that provides a central store for application state.

The middle-tier design part of the application is responsible for processing user requests from the front end and performing necessary business logic to send back a valid response by using RESTful API architecture. The middle tier in our application is developed using NodeJS by following Express framework and RESTful API service.

The backend covers two modules of our application:

User module: User module is responsible for user registration i.e., authentication, signup, and login management. This includes email validations, password validations, user profile management etc.

Establishment module: This module is responsible for managing the establishments in our application such as restaurants, housing and restro-bars. These include APIs for adding, removing, and managing establishments.

Rating module: This module is responsible for managing the ratings given by users to certain establishments. For our application specific functionality, only users from matching ethnicity can rate an establishment to maintain authenticity.

Computer Engineering Department

Batthina, Vineet (MS Software Engineering) Masapeta, Niranjan Reddy (MS Software Engineering)

Namburu, Harshavardhana Reddy (MS Software Engineering)

Rajanala, Guruvardhan Reddy � (MS Software Engineering)

Deployment

We are using horizontal scaling to deploy our application. Horizontal scaling involves deploying applications across multiple servers to ensure that instances can handle multiple requests and increased load. Horizontal scaling ensures our application to achieve high availability by distributing the workload among instances. It is a cost-effective deployment as we don’t need to invest in single high-end hardware to deploy our application. Instead, we have the flexibility to start with smaller hardware setup and incrementally add resources based on the demand.

The web application is deployed on AWS cloud using EC2, along with load balancer and log analytics is deployed on Amazon Kinesis.

• Backend: The backend NodeJS application is deployed onto an EC2 instance on AWS cloud. The security groups are configured for the instance to control inbound and outbound traffic. To ensure security of the application, the access is restricted only to specific IP address ranges configured by elastic IP.

• Frontend: The inbuilt build tool “Create React App” automatically generates production-ready bundle for the frontend application. By using and configuring nginx on the EC2 instance, users can access the frontend application by visiting server’s ip address. Nginx acts as a load balancer to distribute oncoming requests across multiple backend servers.

• Database: Configure the MySQL database in Amazon RDS to ensure seamless integration and data flow between the components. By hosting database on AWS cloud, we are rest assured our data is secure, transactions are reliable and scalable based on demand.

Results/Conclusions

Frameworks

ReactJS: React creates user interface pages for each functionality in the application such as home page, login, signup, establishments listing etc.

NodeJS: NodeJS is a backend runtime environment used to develop backend of our application. Node interacts with our SQL database to perform necessary queries depending on the request from the frontend.

Web application architecture

MySQL: The database is developed using MySQL providing the middle tier architecture to securely connect to the database and perform necessary queries to interact with the tables and retrieve relevant data.