Artur Skrzeta

App Version Tracking with Git

Intro

Where is Human ends is where devops begins:

source: dzone

Git:

• Distributed version control system.
• Coordinates work between multiple developers.
• To see who made what changes and when.
• Local and remote repos.

Git application:

• Keeps track of code history.
• Takes snapshot of the files at the moment.
• You decide when to take a snapshot by making a commit.
• You can visit any snpashot back in time.
• You can stage files before committing.
• Git shouts there are files staged after every change.

Repository:

• Git stores the full history and source control of a project over time.
• Can be hosted on local computer or on a shared server like GitHub
• Contributors wokring on the project make copies of the repository on their local mnachines using the push and pull commands.
  - remote repository --> shared server repository f.e.: GitHub.
  - local repository --> local machine repository.

Workflow:

1. GIT INIT
2. Working directory
3. GIT ADD
4. Staging Area
5. GIT COMMIT
6. Local repository
7. GIT PUSH
8. Remote repository

Features

App includes following features:

• GitHub

Demo

Git on Windows 10:

• Running Git Bash.
• Go to project folder with cd command.
• Create project files as needed with touch command.
• Initialize directory as git repository with git init command:
  - it creates .git hidden folder in the current directory.
• Create user:
  - git config --global user.name 'Artur Skrzeta'
  - git config --global user.email 'arturskrzeta@gmail.com'
• Listing all the configuration you provide
  - git config --list
• Adding a file to stage:
  - git add script.py
• Checking what is in the stage:
  - git status
• Removing files from the stage:
  - git rm --cached script.py
• Adding all files to stage:
  - git add .
  - it commits whole directory paths as well.
• Committing staged files:
  - git commit
  - it opens a new bash window: type 'i' for inserting, 'esc' keyboar for escaping inserting mode, ':wq' for writing and quitting
  
  
• Committing staged files with commentary:
  - git commit -m 'changed script.py'
• Checking all commits:
  - git log
  
  
• Setting a file as ignored to prevent from staging and committing:
  - touch .gitignore
  - in .gitignore type file name to be ingored along with an extension f.e.: index.html
  - in .gitignore type whole directory f.e.: /dir2
  - .gitignore will be staged and committed as well.
• There is another type of a file which is .gitkeep:
  - touch .gitkeep
  - we can use it when we want to keep tracking empty dictionaries,
  - we can then push it to the remote repository,
  - the fact is, the name of it could be anything as there is no special siginificance to the name,
  - example:
  mkdir empty-directory
cd empty-directory
touch .gitkeep
git add .
git commit -m "Commit empty folder in Git with gitkeep"
git push origin
  - to sum it up, we might need to get .gitkeep in the empty directory otherwise we cannot push empty folder to the remote repository.
• With stash we can create a record of a current changes and bring back the clean branch. After that, I can check out other branches or start working from the beginning on the current one:
  git stash
• Removing all from the stage:
  git reset

Branches:

• Commiting changes to project aside without affecting main branch.
• Each branch can be a new functionality that we want to develop without affecting the main brach with the main code base.
• Listing all branches:
  - git branch -a
• Creating a new branch:
  - git branch new_branch
• Switching branches:
  - git checkout new_branch
  
  
• Getting back to master branch makes files created on new_branch disappear:
  - git checkout master
  
  
• Merging master and new_branch branches from master branch:
  - git merge new_branch
• We need to be carefull of merge confilcts.

Remote repository:

• Listing all remote repositories:
  - git remote
• Pushing README.md into GitHub repository
  - git init
  - touch README.md
  - echo "# test" >> README.md
  - git add README.md
  - git commit -m 'Adding readme'
  - git branch -M main
  - git remote add origin https://github.com/ArturSkrzeta/test.git
  - git push -u origin main
  
• Pulling all repository (getting fully synchronized with remote repo):
  - git pull
  - when creating a new branch, in ordert to see it with git branch -a we need to pull remote repo into local one every time we want to by in sync.
• Cloning a repository:
  - git clone https://github.com/ArturSkrzeta/Test2.git
  - copying repository into current directory
• Cloning a repo from branch different than master:
  - git clone -b branch_name remote_repo
• Pushing a new branch to remote repository:
  - git branch new_branch
  - git push origin new_branch
• We can also push repo into CI/CD platform as GitLab:
  git push -u "https://gitlab.com/ArturSkrzeta/my_project.git" master
  - it will ask for your credentials and you have to have an access token set up on the Gitlab account.
  - we need to access token created and assigned to gitlab profile in order to push someting via HTTPS.
  - pushing a new script using ssh:
  git status
git add Deployment_Scripts/v1_test.sql
git commit -m "abc"
git push -u git@....git local_artur
  

GitLab:

• Project forking:
  - copy of a project and it allows to make changes without affecting the original project.
  - steps:
  1. loign into gitlab profile,
  2. enter project,
  3. click fork icon,
  4. if namespace not available, then we go to Groups -> Your groups -> New Group -> name it and create it,
  5. if namespace available, then select it and project forked successfully.
• Securied Shell (SSH):
  - used for authentication, by setting ssh key we can connect to gitlab server without using username and password each time,
  - stpes:
  1. on windows, we need to use putty app or git bash,
  2. running git bash, we need to type the folowing command ssh-keygen and then entering,
  3. once done, in aministrator profile on C drive there is .shh folder created,
  4. then on the gitlab, we need to go to the account's setting > ssh keys > pasting content of id_rsa.pub file > giving it a title.
• CI/CD in GitLab:
  Continuous Integration:
  - dev > app test > integration test
  - multiple devs pushes changes to a common remote repository,
  - every change is built and tested automatically.
  Continuous Delivery:
  - dev > app test > integration test > acceptance test
  - code is checked automatically but deployment needs to be done manually
  Continuous Deployment:
  - dev > app test > integration test > acceptance test > prod
  - deployment is automatic.
• CI/CD in steps:
  1. commit - new code being integrated to the code base.
  2. build - source code is converted to the executable form.
  3. test - checks the interaction between builds and if an app works.
  4. deploy - deploying to the prod env.
• Steps to use Gitlab CI-CD:
  1. App code base hosted on Git repo.
  2. Building and deployment script sequentially defined in the script of YAML fieldset.
  3. YAML file needs to be located in the root path of your repo.
  4. YAML file needs to be called .gitlab-ci.yml.
• CI/CD pipeline:
  source: dzone
  
• YAML configuration file consists of 3 stages:
  source: edureka
  
  and determines:
  - pipeline structure,
  - steps to be executed,
  - condition decisions.

CICD for db:

• Here are following challenges (comparing to App devops):
  source: dzone
  
• Database change management:
  
  

  Imperative	Declarative
  Each change (or set of changes) to db are in their own versioned script taht need to be applied in the right order.	For each object we maintain a one definition file that is being changed over time.
  Requires maintaining state of each db where the vesion has been deployed.	Deploying changes requires a tool creating a migration script (we cannot run the same whole script again as we might already have a table existing). We need a schema comparison tool comparing definition in the code with exisitng database and based on the differences it generates alter statements (or create statemnt if the schema doesn't exist).
  Very felxible but error-prone.	Less error prone.

• Snowchange as database change management (DCM) tool:
  - One of the DCM tools (also known as database migration, schema change management, or schema migration tools).
  - Allows managing Snowflake objects (tables, views, stored procedures, and so on).
  - Flexible, lightweight, written in Python (but nof official Snowflake offering).
  - Combined with a version control system and a CI/CD tool, database changes can be approved and deployed through a pipeline using modern software delivery practices.
  - Implies the imperative way of changes providing.
  - A snowchange change script is just a file that contains one or more SQL statements.
  - Each change script name must include an unique version number (for example, V1.1__first_change.sql) that is used to ensure the scripts are run in the correct order against a target Snowflake database's object.
  - We need to keep to the structure of project folder:

  > project root
    > folder 1
        V1.1.1__first_change.sql
    > folder 2
        V1.1.2__second_change.sql
  

  - we can have any number of folders,
  - in each folder, there is a changing script with a proper naming convention and vesrioning,
  - snowchange starts from the root, walking through every sub-folder recursively and finding all the scripts,
  - we just need to simply run snowchange.py giving it a root project's directory and snowflake credentials.
  
• When running, the snowchange loops through entire database applying any missing changes to the objects.
• Snowchange runs scripts only against the target database with a version number larger than the max version stored in the change history table. This way, only “new” scripts are applied to the target database each time snowchange is run.
• The state of each target database is tracked in a special change history table named SNOWCHANGE.CHANGE_HISTORY (which is automatically created and maintained by snowchange).
• Just to sum up, a DCM tool is critical for managing the Snowflake objects and implementing CI/CD pipelines.
• When merge to UAT or PROD, the cicd pipeline is triggered which runs snowchange script.

CICD deploying changes flow:

• Here is the schema of complete flow of deploying any changes to Snowflake:

1. Developer pulls remote repository from DEV branch in order to be up-to-date with other developers.
2. Developer makes and pushes change scripts from local repository into developer's branch in the remote repository.
3. Developer merges his branch with DEV branch.
4. Developer raises the merge request to UAT branch.
5. Code approver/reviewer accepts the merge request and CICD is triggered.
6. Changes are deployed in Snowflake in UAT environment.
7. Developer raises the merge request to PRD branch.
8. Code approver/reviewer accepts the merge request and CICD is triggered.
9. Changes are deployed in Snowflake in PROD environment.

Setup

Following installation required:

• Git installation from https://git-scm.com/

Source Code

You can view the source code: HERE