Handling Data in Steps in Data Pipelines

Intro

Data Pipeline is a series of following after each other steps of data processing. Within a series, each step delivers an output that is the input to the next step.

Features

App includes following features:

  • RESTful API
  • Flask
  • SQLite

Demo

Workflow:
  1. Company A generates and streams data to a web server.
  2. At the same time Company B requests a web server for a data stream with GET method.
  3. Company B handles the inflowing data with programmed business logic.
  4. Company B has database on its side and feeds it with inflowing data.

I can send the GET request through a web-browser as a client getting data from web server:

The same get request is being sent by the Company B.

Setup

Python libraries installation required:

pip install flask
pip install sqlite3

Source Code

You can view the source code: HERE