ishenwei/nexus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b40abbcd473a7093d8261e212e3d6de97c1e516a
nexus/knowledgebase/csd-wiki/ICSD/How-to-replace-bastion-with-Rocky-Linux_688996309.md

7.2 KiB
Raw Blame History

How-to-replace-bastion-with-Rocky-Linux_688996309

Most of current linux bastion are EOL(end of life), security team is pushing to use new ones seriously, here is the steps that you can refer to

Please replace texts that is Italic and wrapped with <> to your own

  • Launch EC2 & configure SSM connect
    • Create a new CCoE-Rocky bastion node by coping the settings from the previous bastion node:
      • Navigate to EC2 → select the previous bastion node instance → Actions → Image and templates → Launch more like this - "Browse more AMIs" and select CCoE-RockyLinux9-x64-2024-07-24 in "My AMIs" → "Shared with me"; Confirm changes. - Make sure the settings are the same like previous bastion (especially for the Key pair, SG, instance type, Subnets, tags). Change the storage to 100GB gp3.
        • For some farms, there are 2 disks on bastion, in most case the second one is for special case but forgot to delete, you can keep only the primary disk for the new bastion
      • Update IAM role(Action → Security → Modify IAM role) to the same one as old bastion, stop then start EC2
      • Now try to access the new bastion - ssh user changed to rocky(compared to centos before)
      • If you can not connect to new bastion, upgrade your terminal (putty or mremoteNG or...)
  • yum install required software(for the first time to yum install, it takes time, after that it's much faster)
  • pip install required software
    • python -m pip install boto3 XlsxWriter kubernetes psycopg2-binary
      • other python module used by you
  • aws/kubectl/eksctl
    • curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/1.29.3/2024-04-19/bin/linux/amd64/kubectl && cp kubectl /usr/local/bin/ && cp kubectl /usr/bin/ && chmod a+x kubectl /usr/local/bin/kubectl /usr/bin/kubectl
      • curl --silent --location " [https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_](https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_) (uname -s)_amd64.tar.gz" | tar xz -C /tmp && echo y | cp /tmp/eksctl /usr/local/bin/ && echo y | cp /tmp/eksctl /usr/bin/ && chmod a+x /usr/local/bin/eksctl /usr/bin/eksctl && echo -e "\n"$(eksctl version)
      • Copy files from ~/.aws/ to new bastion(same folder structure & same permissions otherwise chmod), then test using aws s3 ls --profile us2-prod (some jenkins job will reach to us2-prod) - better to do for both root & rocky user(you can sudo su - to the root user first and then su rocky)
      • Copy file: ~/.kube/config to new bastion(same folder structure & same permissions otherwise chmod), then test using kubectl get no - should return same output as old bastion - better to do for both root & rocky user(you can sudo su - to the root user first and then su rocky)
  • jq/yq/helm - can be from OMT installtion package
    • Download current OMT package to new bastion(under /opt/, since some jenkins jobs search for that path), unzip and change directory to the folder
      • Check current OMT version by running helm commands on old bastion: helm ls -n core | awk '{print $10}'
      • ./install --capabilities Tools=true,Monitoring=false,LogCollection=false,DeploymentManagement=false,ClusterManagement=false
      • ignore the warning
      • Update /root/itom-cdf.sh, export CDF_NAMESPACE=core
      • cp /root/cdf/bin/helm /usr/local/bin/ && cp /root/cdf/bin/helm /usr/bin/ && chmod a+x /usr/local/bin/helm /usr/bin/helm
      • run some cdf scripts to test(if you can not use that, disconnect and reconnect)
      • yq --version - jq --version - helm ls -n core - sh refresh-ecr-secret.sh -r < region-code>
  • Velero
    • velero client config set namespace=velero
      • cp /root/cdf/bin/velero /usr/local/bin/ && cp /root/cdf/bin/velero /usr/bin/ && chmod a+x /usr/local/bin/velero /usr/bin/velero
      • Set velero backup per 6 hours: velero schedule create velero-< us2-dev> -cron --ttl 168h0m0s --schedule="0 3/6 * * *"(avoid doing this if you already did that in other bastions, check by velero schedule get)
  • mount EFS
    • cd /mnt && mkdir -p efs && mkdir -p cms && mkdir -p oomt && mkdir -p monitoring && chmod o+r cms/ efs/ monitoring/ oomt/ && chmod o+x cms/ efs/ monitoring/ oomt/ (note there might be different path for different farms)
      • Update /etc/fstab, Add 4 efs server as 4 lines into that file(better to copy from old bastion)
  • Other Settings
    • systemctl enable qualys-cloud-agent && systemctl start qualys-cloud-agent
      • Change the file content: /etc/hostname to farmbastion, e.g. < us2dev >bastion
      • Copy specific lines in /etc/hosts from old bastion to the same file in new bastion (depends on you)
      • Check if there is any cron jobs running on the old bastion, you can copy to the new bastion(and related scripts) if you think it's valid - crontab -l (disable previous one, make sure no duplicate job running in both bastion at the same time)
      • Change umask from 0027 to 0022 for both root and rocky user(Fix permissions of new file and directory not the same as old bastion) by: echo 'umask 0022' >>~/.bashrc
      • echo 'source <(kubectl completion bash)' >>~/.bashrc
      • echo 'source <(helm completion bash)' >>~/.bashrc
      • Test settings after EC2 restarts:
      • df -h, check the 4 efs server should be mounted - hostname should be updated to bastion - type umask, should return 0022 - kubectl get po -n itsma + tab - should auto fill the right namespace - helm ls -n itsma + tab - should auto fill the right namespace
        • Make sure you don't see the warning: WARNING: Kubernetes configuration file is group-readable. This is insecure.
  • Post tasks
    • Update values in parameter store (search by keyworkds: bastion1, and then update IP to the new bastion IP and login user to rocky)
      • Test by running jenkins job on new bastion
      • Copy /root/worknodes.pem from old bastion to new bastion(under /root, this will be used for qualys agent installation on EKS worknodes) and change the permission: chmod 400
      • Shutdown the old bastion(you can terminate it after it's stable)
      • Notify team about the new bastion - they will update the bat commands

Related pages

Reference in New Issue View Git Blame Copy Permalink