What is this post about?

Couple of days ago some colleagues are complaining about unbearable performance of SAP Application which is currently host with OpenSUSE on Azure VM, shortly after my manager call me in on Monday morning and tells me to reboot production environment of SAP.

Even though I know something will hit the fan because I surely did remember last time we reboot that machine, is 700+ days before.

As expected VM stuck at closing db process for an hour and a half, after briefing my manager, he said "Hard reset." and it died, completely.

I tried to stop it but NooOoOOooOOoo they don't listen to me!

So this post is about how I spent 26 hours only depend on Microsoft messy documentation and try and error.

Just in case I won't step into this mud again.

CAUTION

Perform such things like hard reset during a linux shutdown process will highly likely to damange your boot partition.

Diagnose Problem

On Azure Portal find your VM and check the boot log by click the Serial Console on the left.

If message like down below appears, you might have yourself a broken boot partition.

Failed to start File System Check on /dev/disk/by-uu...d-121f462d7e8d

Solution

Prerequisite

Install az cli extension

Refer Installation Guide

Install vm-repair Extension

If you haven't install this extension before.

$ az extension add -n vm-repair

If you already installed this extension, it's always a good idea to check extensions update.

$ az extension update -n vm-repair

Setup Repair VM

$ az vm repair create -g {{MyResourceGroup}} -n {{myVM}} --repair-username {{username}} --repair-password {{password!234}} --verbose

What this command will act is copy system partition files and mount it to a new VM which is automaticaly created.
Wait for couple of minutes and you can ssh into your new VM.

Start Repair

Repair Command

SSH into your repair VM, use $ lsblk to check the device Id of your broken parition and run:

$ fsck /dev/{{device_name}}

Y on all questions.

Unattened Script

Azure also provided a automatic repair script but I haven't tried it.

$ az vm repair run –g {{MyResourceGroup}} –n {{MyVM}} -–run-on-repair --run-id 2 --verbose

Complete and Restore

Delete Repair VM

Use command down below to delere your repair VM.

$ az vm repair restore -g {{MyResourceGroup}} -n {{MyVM}} --verbose

This will replace the partition we just fixed to broken partition.

And depends on your needs you can choose to keep or delete your repair VM, but it will still be charged even it's not booted.

Boot up VM

Now you can boot up your VM with fixed boot partition.