Python script to clone a replica of production website and host it as a sub domain.

 Steps to manually create a replica of production website and host it as a sub domain.

  1. In order to create a replica of production website we need to create a DNS by adding the ip address of the server where you would like to host in your Domain DNS service provider.
  2. Create a test database by importing the backup database dump file of the production website
    1. Also grant all the privileges of the new database to the MySQL user
    2. Flush Privileges
  3. Clone the code base (repository) into your “www ” folder on the server
    1. Fix permissions if required
  4. Add the robots.txt (which disallows the crawler) file to the new cloned folder (In order to make this test website hide from the web crawlers, so that it will not appear in search engines)
  5. Add the Nginx conf file to the nginx/conf.d folder
  6. Restart the Nginx.

 

Now implementing all these steps using a python script and executing them using a command line

GitHub link for the python script

 This repository contains 3 shovel tasks
  1. create_test_db_from_backupfile — Creates a new test database from backup sql file
  2. clone_new_dev_instance — Clone the project repository on the web server.
  3. teardown_dev — Delete the testing website and its related conf files in case you don’t need it (Optional task in case you need to completely delete the replicated website one may use this task)
Prerequisites for this python script to work
  1. Unix based web server which also includes nginx
  2. Python 2.7 Installed
  3. Install shovel [Instructions to install Shovel](https://github.com/seomoz/shovel#installing-shovel)

How to run the replica of production website script

  1. Add the sub domain on which you wish to create the replica website in DNS Service and link in to the the static IP of the web server

 

 

To know more about the author visit  cvarma.com to read other articles click here

Migrate GitLab-Mattermost to new server

GitLab is a open source application to maintain git repositories. Mattermost is a private cloud messaging and an alternate for Slack. The following blog post is specifically for instructions to migrate Gitlab-Mattermost to new server. So assuming you have basic knowledge about Gitlab-Mattermost service if not visit this links here GitLab-Mattermost or you may also visit this official blog post on GitLab-Mattermost.Image of the fish leaping from one bowl to another, which is similar to migrate Gitlab-Mattermost

Steps in brief to migrate GitLab-Mattermost to new server

  1. Install GitLab, enable HTTPS using Let’s Encrypt and check its version on the new server
  2. Update the current GitLab instance running on the current server
  3. Create a gitlab backup on your current server and upload to AWS S3 or Cloud Storage in GCP
  4. Take a backup for gitlab.rb file
  5. Download the backup from AWS S3 or Cloud Storage in GCP
  6. Restore the Backup.
  7. Make sure GitLab Settings in Mattermost’s config.json matches with config.json on old server
  8. Finally Switch the DNS to test Gitlab-Mattermost Migration.

Steps in Detailed to migrate GitLab-Mattermost to new server

(Note all the below mentioned commands are for Ubuntu 16.04. But almost the process and steps remains same irrespective of OS. So depending on your OS, refer Google for exact commands to that corresponding step)

Install Gitlab, enable HTTPS using Let’s Encrypt and check GitLab version on the new server

$>sudo apt-get update
$>sudo apt-get install curl openssh-server ca-certificates postfix curl -sS
$>https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
$>sudo apt-get install gitlab-ce
$>sudo gitlab-ctl reconfigure
$>sudo gitlab-rake gitlab:env:info (# One can find the Gitlab version under the Gitlab Information)

One may also use official GitLab installation documentation to install GitLab, Note: Mattermost comes by default in GitLab, we need to enable Mattermost in GitLab.

One may use Digital Ocean’s article on How To Secure GitLab with Let’s Encrypt on Ubuntu 16.04  in order to enable HTTPS for GitLab, one may also choose different article to secure GitLab with Let’s Encrypt depending on OS of your Web Server.

Update the current GitLab instance running on the current server

Run  the following command in order to find the GitLab version on the current server

$>sudo gitlab-rake gitlab:env:info

If you find the version on the current server doesn’t match with the new server, then run the following command and update the current GitLab version on the current server.

$>sudo apt-get update && sudo apt-get install gitlab-ce

Create a GitLab backup on your current server and upload to AWS S3 or Cloud Storage in GCP

In order to take a backup and we need to configure the gitlab.rb file based on the cloud storage service you use.

Follow this official Uploading backups to a remote (cloud) storage documentation in order to modify the gitlab.rb (configuration file) depending on your remote (cloud) storage

I have used AWS S3 and my modification are as follows

$>sudo vi /etc/gitlab/gitlab.rb

I have the added the following lines at the end of the gitlab.rb file

gitlab_rails['backup_upload_connection'] = {
 'provider' => 'AWS',
 'region' => 'eu-west-1',
 'aws_access_key_id' => 'AKIAKIAKI',
 'aws_secret_access_key' => 'secret123'
 }
 gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket'

After modifying gitlab.rb run $>sudo gitlab-ctl reconfigure in order to make the changes effective

Create backup of gitlab on current server

$>/opt/gitlab/bin/gitlab-rake gitlab:backup:create

Take backup for gitlab.rb file and Mattermost’s config.json file

Take backup of gitlab.rb file (location for gitlab.rb file /etc/gitlab/gitlab.rb) into your local computer

Sign is as Mattermost by using this command

$>sudo -u mattermost -i bash

Take the backup for Mattermost config.json file into your local computer. (open config.json file in home folder on signing as Mattermost).

Download the backup from AWS S3 or Cloud Storage in GCP

One have to download the backup from remote cloud storage into the new server and copy it into the GitLab backups folder (/var/opt/gitlab/backups/ folder) on the new server.

If you are using AWS S3 as remote cloud storage service, then you may follow the following steps in order to download and copy GitLab Backup into GitLab backups folder.

Install AWS CLI and configure AWS CLI

$>sudo apt-get install awscli

$>aws configure

Copy the backup file and copy it into the /var/opt/gitlab/backups/ folder

$>aws s3 cp s3://s3-bucket-name/[TIMESTAMP]_gitlab_backup.tar /var/opt/gitlab/backups/[TIMESTAMP]_gitlab_backup.tar

Replace the gitlab.rb file on the new server and Restore the Backup.

Stop all the services in order to restore the Gitlab-Mattermost Date

$>sudo gitlab-ctl stop unicorn

$>sudo gitlab-ctl stop sidekiq 

$>sudo gitlab-ctl status (To check the status)

Restore the Data from the backup file

$>sudo gitlab-rake gitlab:backup:restore BACKUP=[TIMESTAMP]

Replace the gitlab.rb file on the new server with the old gitlab.rb file stored in your local.(back up from the old server)

Reconfigure the gitlab and start the server

$>sudo gitlab-ctl reconfigure

$>sudo gitlab-ctl start

$>sudo gitlab-rake gitlab:check SANITIZE=true

Make sure GitLab Settings in Mattermost’s config.json matches with config.json on old server

mainly go through the following settings in config.json and make sure they match (GitLab setting in config.json are so important to match if you have GitLab as authentication agent for your Mattermost)

"GitLabSettings": {
 "Enable": true,
 "Secret": "[Secret_Key]",
 "Id": "[ID]",
 "Scope": "",
 "AuthEndpoint": "https://gitlab.[domainname].com/oauth/authorize",
 "TokenEndpoint": "https://gitlab.[domainname].com/oauth/token",
 "UserApiEndpoint": "https://gitlab.[domainname].com/api/v3/user"

In case if they are not matching open gitlab.rb  and edit the following settings and reconfigure gitlab-ctl

mattermost['gitlab_enable'] = true
mattermost['gitlab_id'] = "[ID]"
mattermost['gitlab_secret'] = "[Secret_Key]"
mattermost['gitlab_scope'] = ""
mattermost['gitlab_auth_endpoint'] = "https://gitlab.[domainname].com/oauth/authorize"
mattermost['gitlab_token_endpoint'] = "https://gitlab.[domainname].com/oauth/token"
mattermost['gitlab_user_api_endpoint'] = "https://gitlab.[domainname].com/api/v3/user"

make sure the above mentioned settings are not commented in gitlab.rb file, change the [ID], [Secret_Key], [domainname] according to config.json file on Old Server.

$> sudo gitlab-ctl reconfigure

Finally Switch the DNS to test GitLab-Mattermost Migration.

Finally change the DNS for Gitlab and Mattermost sub domains of your domain to the new server IP address

If everything goes well you would have been successfully Migrated your GitLab-Mattermost to new server.

Happy Migrate Gitlab-Mattermost 🙂

To know more about the author visit  cvarma.com to read other articles click here