V4 storage type DEEP DIVE

 

MAOR lipchuk

Senior software Engineer

mlipchuk@redhat.com

AGENDA

  • UPGRADE PROCESS - API/GUI
  • VERIFY UPGRADE - API/GUI/LOGS/METADATA
  • QCOW2 VERSION - COMPATIBILITY VERSION
  • NEW API / NEW COMMANDS
  • QCOW2 VOLUME UPGRADE - COMMANDS & LOGS
  • TROUBLESHOOTING
  • Q/A

UPGRADE PROCESS

STEP 1 - UPGRADE CLUSTERS

REST

GUI

PUT http://localhost:8080/ovirt-engine/api/clusters/123 HTTP/1.1
Accept: application/xml
Content-type: application/xml

<cluster>
  <version>
     <major>4</major>
     <minor>1</minor>
  </version>
</cluster>

UPGRADE PROCESS

STEP 2 - UPGRADE DATA CENTER

REST

GUI

PUT http://localhost:8080/ovirt-engine/api/datacenters/111111 HTTP/1.1
Accept: application/xml
Content-type: application/xml

<data_center>
  <version>
     <major>4</major>
     <minor>1</minor>
  </version>
</data_center>

UPGRADE PROCESS

  • No changes in the upgrade process
  • ISO and Export Storage Domain will not be reflected.
  • Data Storage Domains will be upgraded to version 4:  QCOW2 version 3 support
  • All new QCOW volumes will be created with compatibility level of 1.1
  • Existed QCOW volumes will have 0.10 comp level.
  • DB -

           1) Clusters and Data Center will be updated to 4.1

           2) new field called qcow_compat in images table

UPGRADE PROCESS

VERIFY DC VERSION - REST

UPGRADE PROCESS

VERIFY DC VERSION - GUI

UPGRADE PROCESS

VERIFY LOGS

... RPC call StoragePool.getInfo succeeded in 0.21 seconds (__init__:522)
... Run and protect: upgradeStoragePool(spUUID='99d3700b-fcb9-44fe-a29c-f83217a9ab75', targetDomVersion='4') (logUtils:51)
... Trying to upgrade master domain `05d36ad9-960c-453d-865e-88fdcb5e60e9` (sp:451)
... Creating external leases volume /rhev/data-center/mnt/10.35.16.43:_export_data6/05d36ad9-960c-453d-865e-88fdcb5e60e9/dom_md/xleases (fileSD:754)
... Formatting index for lockspace u'05d36ad9-960c-453d-865e-88fdcb5e60e9' (version=1) (xlease:651)

... Run and protect: upgradeStoragePool, Return response: {'upgradeStatus': 'started'} (logUtils:54)
... [jsonrpc.JsonRpcServer] RPC call StoragePool.upgrade succeeded in 0.36 seconds (__init__:522)

HOST LOGS - upgradeStoragePool

ENGINE LOGS - UpgradeStoragePoolVDSCommand

[org.ovirt.engine.core.bll.storage.pool.UpdateStoragePoolCommand] (org.ovirt.thread.pool-5-thread-32) [edc48f04-e125-4216-9068-a5bde76c1bdd] Running command:
 UpdateStoragePoolCommand internal: false. Entities affected :  ID: 99d3700b-fcb9-44fe-a29c-f83217a9ab75 Type: StoragePoolAction group EDIT_STORAGE_POOL_CONFIGURATION with role type ADMIN
[org.ovirt.engine.core.bll.storage.pool.UpdateStoragePoolCommand] (org.ovirt.thread.pool-5-thread-32) [edc48f04-e125-4216-9068-a5bde76c1bdd] Setting storage 
domain '05d36ad9-960c-453d-865e-88fdcb5e60e9' (type 'Master') to format 'V4'
[org.ovirt.engine.core.vdsbroker.irsbroker.UpgradeStoragePoolVDSCommand] (org.ovirt.thread.pool-5-thread-32) [edc48f04-e125-4216-9068-a5bde76c1bdd] START, Up
gradeStoragePoolVDSCommand( UpgradeStoragePoolVDSCommandParameters:{runAsync='true', storagePoolId='99d3700b-fcb9-44fe-a29c-f83217a9ab75', ignoreFailoverLimit='false', storagePoolId='99d3700
b-fcb9-44fe-a29c-f83217a9ab75', poolVersion ='4'}), log id: 2d353c4c
[org.ovirt.engine.core.vdsbroker.irsbroker.UpgradeStoragePoolVDSCommand] (org.ovirt.thread.pool-5-thread-32) [edc48f04-e125-4216-9068-a5bde76c1bdd] Upgrading
 storage pool '99d3700b-fcb9-44fe-a29c-f83217a9ab75' to version '4'.
[org.ovirt.engine.core.vdsbroker.irsbroker.UpgradeStoragePoolVDSCommand] (org.ovirt.thread.pool-5-thread-32) [edc48f04-e125-4216-9068-a5bde76c1bdd] FINISH, U
pgradeStoragePoolVDSCommand, log id: 2d353c4c

UPGRADE PROCESS

VERIFY  METADATA

Storage Type: Block Storage Domain

 

Grep The Metadata Using Command Line:

 

 

 

Example:

 

 

sudo vgs -o +tags

UPGRADE PROCESS

VERIFY METADATA

Storage Type: File Storage Domain

 

Metadata File location:

 

 

 

Example:

 

 

 

 /rhev/data-center/mnt/11.11.11.11:_export_data3/11111/dom_md

QCOW2

 

  • The QCOW image format is one of the disk image formats supported by the QEMU processor emulator.

  • QCOW3 (QCOW2V3) is strictly an extension of QCOW2 and keeps the fundamental structure unchanged, so that a single codebase will be enough for working with both QCOW2 and QCOW3 images

 

  • The QCOW image format includes a version number to allow introducing new features that change the image format in an incompatible way. Newer QEMU versions (1.7 and above) support a new format of QCOW2 version 3, which is not backwards compatible and introduce many improvements like zero clusters, and performance improvement.
  •  
  • Easy to upgrade from QCOW2 to QCOW3

SPECIFICATION FOR QCOW2V3

Based on http://git.qemu-project.org/?p=qemu.git;h=4fabffc1

- Added compatible/incompatible/auto-clear feature bits


- Configurable refcount width.


- Zero cluster flags.

- Fixed internal snapshot metadata to use 64 bit VM state size.

- Extended internal snapshot metadata to contain the disk size

 

 

 

QCOW2 VOLUME COMPATIBIILITY VERSION

qemu-img info

-sh-4.2$ qemu-img info test.qcow2 
image: test.qcow2
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 196K
cluster_size: 65536
Format specific information:
    compat: 0.10
    refcount bits: 16

QCOW2 VOLUME COMPATIBILITY VERSION

REST API

GET http://localhost:8080/ovirt-engine/api/storagedomains/123/disks/111 HTTP/1.1
Accept: application/xml
Content-type: application/xml
....
<alias>VM_Disk1</alias>
<format>cow</format>
<image_id>2baf2249-14ac-411b-9392-f97fd80a1c2d</image_id>
<propagate_errors>false</propagate_errors>
<provisioned_size>1073741824</provisioned_size>
<qcow_version>qcow2_v3</qcow_version>
<shareable>false</shareable>
<sparse>true</sparse>
<status>ok</status>
<storage_type>image</storage_type>
....
</disk>

OUTPUT:

QCOW2 VOLUME COMPATIBILIY VERSION

DB

engine=> SELECT image_group_id, image_guid, qcow_compat, volume_format
engine=> FROM images
engine=> ORDER BY image_group_id
          image_group_id             |              image_guid              | qcow_compat | volume_format 
-------------------------------------+--------------------------------------+-------------+---------------
011b693d-2d10-44e5-88ca-fc07536e09bf | c40a49ae-5885-46dc-86e8-2076285581f0 |           0 |             5
5a6630f1-7b6d-4a9f-9d09-8c0f551ec652 | f2a20bed-9240-49a3-aa0d-a3bc85d583d7 |           1 |             4
5a6630f1-7b6d-4a9f-9d09-8c0f551ec652 | 08cc1260-f175-4ee4-9319-96862dbe9005 |           1 |             4
84ab8933-dab4-4caf-82f3-38ac27452adb | 0c863cfa-4d28-4015-b615-546bc17f8231 |           0 |             5
84ab8933-dab4-4caf-82f3-38ac27452adb | ef6b2b68-f609-4c31-b6e3-209e211bc743 |           2 |             4
84ab8933-dab4-4caf-82f3-38ac27452adb | 20fe8e9f-f405-42fa-b96a-f929be688100 |           2 |             4
84ab8933-dab4-4caf-82f3-38ac27452adb | 0f14e0bd-a26d-4b56-8b6e-0fa8d6a3c7b7 |           2 |             4
a994ad57-d9c6-4b14-aebc-ff907d56cd6c | 2e2adfa0-a710-421a-965d-a191da289285 |           0 |             5
e9943fb1-efaa-4d91-bbf4-74a762e3fa12 | f3aa0aa3-9700-4045-b059-7cdc6ff8a516 |           0 |             5
e9accc47-72e1-4daf-bbf7-573c3b76a325 | 623656d6-5788-443c-beb6-3a43e2a0103c |           2 |             4
fe87e8ca-b7b8-48e2-803b-a065be7fbd5d | d122d742-3de3-4731-8a2c-7c453e7e8c53 |           0 |             5

OUTPUT:

NEW VDSM API

  • AMEND_VOLUME - support amending an old QCOW compat to a new compat level using qemu-img amend.

 

  • qemu-img amend can be used to upgrade or downgrade existing images.

 

  • GETQCOWVOLUMEINFO - Returns the output with the compatibility level of the volume by qemu-img info

 

 

NEW ENGINE COMMANDS

  • AmendImageGroupVolumesCommand - Start the operation on the disk
  • AmendVolumeCommand - calls the amend command in VDSM

Engine

QCOW2 VOLUME UPGRADE

-sh-4.2$ qemu-img info test.qcow2
image: test.qcow2
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: 196K
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
-sh-4.2$ qemu-img amend -o compat=1.1 test.qcow2

qemu-img - The amend operation

QCOW2 VOLUME UPGRADE

REST API

PUT http://localhost:8080/ovirt-engine/api/storagedomains/123/disks/111 HTTP/1.1
Accept: application/xml
Content-type: application/xml

<disk>
    <qcow_version>qcow2_v3</qcow_version>
</disk>

QCOW2 VOLUME UPGRADE

QCOW2 VOLUME UPGRADE

LOGS

 

VDSM

Run and protect: sdm_amend_volume(job_id=u'93ad75e5-963d-4382-9050-b90284bac2b2', vol_info={u'generation': 1, u'img_id': u'5a6630f
1-7b6d-4a9f-9d09-8c0f551ec652', u'sd_id': u'ab7fef7c-8992-4797-8c62-4030ba284b1f', u'vol_id': u'c53990f8-1512-4b0c-ade3-f85e3d467173'}, qcow2_attr={u'compat': u'1.1'}) (logUtils:51)
Run and protect: sdm_amend_volume, Return response: None (logUtils:54)
RPC call SDM.amend_volume succeeded in 0.00 seconds (__init__:516)
START task 8e155f70-34d1-4c50-9083-04bc56bd566b (cmd=<bound method Task.commit of <vdsm.storage.task.Task insta
nce at 0x3828b48>>, args=None) (threadPool:208)

ENGINE

[org.ovirt.engine.core.bll.storage.disk.image.AmendImageGroupVolumesCommand]
 (DefaultQuartzScheduler9) [4ddf9b3b-7d22-4d0b-a4b7-3eb364320899] Starting child command 1 of 5, image '4bf9db37-1975-4d68-b885-
e8bb87ad99bf'
[org.ovirt.engine.core.bll.storage.disk.image.AmendVolumeCommand] (DefaultQuartzScheduler9) [10175c90] Running command: AmendVolumeCommand
 internal: true.
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (DefaultQuartzScheduler9) [10175c90] START, 
GetVolumeInfoVDSCommand(HostName = venus-vdsa.tlv.redhat.com, GetVolumeInfoVDSCommandParameters:{runAsync='true', hostId='ae0a04a6-6e2d-
4225-a5ed-e9b2418119eb', storagePoolId='7509df80-6037-4070-a48d-252742b82921', storageDomainId='ab7fef7c-8992-4797-8c62-4030ba284b1f', 
imageGroupId='5a6630f1-7b6d-4a9f-9d09-8c0f551ec652', imageId='4bf9db37-1975-4d68-b885-e8bb87ad99bf'}), log id: 3b7a9106
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (DefaultQuartzScheduler9) [10175c90] FINISH, GetVolumeInfoVDSCommand, 
return: org.ovirt.engine.core.common.businessentities.storage.DiskImage@c6090116, log id: 3b7a9106
[org.ovirt.engine.core.vdsbroker.vdsbroker.AmendVolumeVDSCommand] (DefaultQuartzScheduler9) [10175c90] START, 
AmendVolumeVDSCommand(HostName = venus-vdsa.tlv.redhat.com, AmendVolumeVDSCommandParameters:{runAsync='true', hostId='ae0a04a6-6e2d-4225-
a5ed-e9b2418119eb', storageDomainId='ab7fef7c-8992-4797-8c62-4030ba284b1f', jobId='6a12cda7-63e0-4378-ade3-b899baeed3bd', 
imageId='5a6630f1-7b6d-4a9f-9d09-8c0f551ec652', volumeId='4bf9db37-1975-4d68-b885-e8bb87ad99bf', generation='1', qcowCompat='QCOW2_V3'}), 
log id: 4110968d
[org.ovirt.engine.core.vdsbroker.vdsbroker.AmendVolumeVDSCommand] (DefaultQuartzScheduler9) [10175c90] FINISH, AmendVolumeVDSCommand, log 
id: 4110968d
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (DefaultQuartzScheduler9) [10175c90] START, 
GetHostJobsVDSCommand(HostName = venus-vdsa.tlv.redhat.com, GetHostJobsVDSCommandParameters:{runAsync='true', hostId='ae0a04a6-6e2d-4225-
a5ed-e9b2418119eb', type='storage', jobIds='[6a12cda7-63e0-4378-ade3-b899baeed3bd]'}), log id: 3590f1e2
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (DefaultQuartzScheduler9) [10175c90] FINISH, GetHostJobsVDSCommand, 
return: {6a12cda7-63e0-4378-ade3-b899baeed3bd=HostJobInfo:{id='6a12cda7-63e0-4378-ade3-b899baeed3bd', type='storage', 
description='amend_volume', status='done', progress='null', error='null'}}, log id: 3590f1e2
[org.ovirt.engine.core.bll.StorageJobCallback] (DefaultQuartzScheduler9) [10175c90] Command AmendVolume id: 'e58dd438-9d84-440d-afe6-
cca83b096792': job '6a12cda7-63e0-4378-ade3-b899baeed3bd' execution was completed with VDSM job status 'done'
[org.ovirt.engine.core.bll.StorageJobCallback] (DefaultQuartzScheduler9) [10175c90] Command AmendVolume id: 'e58dd438-9d84-440d-afe6-
cca83b096792': execution was completed, the command status is 'SUCCEEDED'
[org.ovirt.engine.core.bll.storage.disk.image.AmendVolumeCommand] (DefaultQuartzScheduler10) [10175c90] Ending command 
'org.ovirt.engine.core.bll.storage.disk.image.AmendVolumeCommand' successfully.
[org.ovirt.engine.core.bll.storage.disk.image.AmendImageGroupVolumesCommand] (DefaultQuartzScheduler8) [10175c90] Starting child command 2 
of 5, image 'a37f9bd0-7337-4d12-9f29-884a161fb8b3'

TROUBLESHOOTING

 

  • Check the DB for the volume format and QCOW version with the following command:

 

 

 

  • Once the volume is created you can check the QemuImageInfo call in the vdsm.log on the Host. the call should indicate the format of the volume and its compat, it should look like this:

 

 

 

 

 

In this example the volume that is returned is with compat 1.1 and the format is QCOW2.

SELECT image_guid,volume_type,volume_format,qcow_compat
FROM images;
2016-12-14 23:18:53,818 INFO (jsonrpc/4) [dispatcher] Run and protect:
 getQemuImageInfo, Return response: {'info': {'compat': u'1.1', 'clustersize': 65536,
 'backingfile': u'bc7a3e50-a965-45 df-8699-ac47a0e20aa5', 'virtualsize': 1073741824,
 'format': u'qcow2'}} (logUtils:52)

LINKS

QCOW2 compat levels - oVirt:

Features/Qcow3 - QEMU:

http://wiki.qemu-project.org/Features/Qcow3
http://www.ovirt.org/develop/release-management/features/storage/qcow2v3/

THANK YOU!

storage type

By Maor Lipchuk