[Windows 8] Add people tagging feature to your Windows Store application !

November 5th, 2012 | Posted by Tom in .NET | Article | C++ | Windows 8 | WinRT | Read: 2,472

People tagging is a feature that can be found in a lot of applications that manipulate pictures (like Windows Live Photo Gallery, Picasa, etc.) but it’s also used on some well-know web applications like Facebook !

Today, we’ll see how to add this feature in your Windows Store application. The code will demonstrate how to tag people on pictures but also how to retrieve the persons tagged on a picture. So, if you plan to create an application that use/display photos, you may want to look at this article to add the people tagging feature.

Note: Just a quick note about the quality of the code. As you’ll see, a bit of C++ is required and I’m real beginner about C++ so please, forgive any mistakes/errors in C++ code and just remember that code is simple a POC that you could improve by yourself !

Introduction about People Tagging

People Tagging is a feature documented on MSDN under the WIC (Windows Imaging Component) part. Indeed, to be able to tag people on images, you’ll need to access metadata and this can be done only with WIC. In .NET, there is a class, called BitmapMetadata, which allows you to access bitmap metadata. Unfortunately, this class is not available in Windows 8 so we’ll need to use C++ to access WIC and expose the feature.

Microsoft has created a new schema, Extensible Metadata Platform (XMP) for tagging people within a digital image. This schema enables applications to store the names and locations of individuals who are in the image as metadata within the image. In addition to the new schema, the new photo property System.Photo.PeopleNames is available in Windows 7. This new property enables applications to read the individual’s names stored in the image metadata. WIC utilizes these new features by enabling applications to easily read and write people tagging metadata to digital photos.

With the name, developers can also retrieve another property, Rectangle, that define the coordinates corresponding to the people tagged in a picture.

Here is a simple representation of the XMP metadata for people tagging:

image

Implementation

To read and write metadata such as the new people tagging feature, we need to use the IWICMetadataQueryReader and IWICMetadataQueryWriter interfaces. These interfaces enable applications to use the metadata query language to write metadata to the individual frames of an image.

To add metadata, the simplest (but longer) way is to re-encode the image and add the metadata (so, in fact, you just create a copy of the image and add the new metadata). As WIC s not accessible in C# (except using some tools like SharpDX, an open-source project delivering the full DirectX API under the .NEt platform), we’ll see how to do that using C++:

void TagMeManager::AddPeopleNameToFile(Platform::String^ sourcePath, Platform::String^ copyPath, Platform::String^ peopleName, Platform::String^ peopleCoordinates)
{
    IWICImagingFactory *piFactory = NULL;
    IWICBitmapDecoder *piDecoder = NULL;

    // WIC Factory
    Tools::Check(
        CoCreateInstance(
        CLSID_WICImagingFactory,
        nullptr,
        CLSCTX_INPROC_SERVER,
        IID_PPV_ARGS(&piFactory))
        );

    Tools::Check(piFactory->CreateDecoderFromFilename(
        sourcePath->Data(),
        NULL,
        GENERIC_READ,
        WICDecodeMetadataCacheOnDemand, //For JPEG lossless decoding/encoding.
        &piDecoder)
        );

    // Variables used for encoding.
    IWICStream *piFileStream = NULL;
    IWICBitmapEncoder *piEncoder = NULL;
    IWICMetadataBlockWriter *piBlockWriter = NULL;
    IWICMetadataBlockReader *piBlockReader = NULL;

    WICPixelFormatGUID pixelFormat = { 0 };
    UINT count = 0;
    double dpiX, dpiY = 0.0;
    UINT width, height = 0;

    // Create a file stream.
    Tools::Check(piFactory->CreateStream(&piFileStream));

    // Initialize our new file stream.
    Tools::Check(piFileStream->InitializeFromFilename(
        copyPath->Data(),
        GENERIC_WRITE)
        );

    // Create the encoder.
    Tools::Check(piFactory->CreateEncoder(
        GUID_ContainerFormatJpeg,
        NULL,
        &piEncoder)
        );

    // Initialize the encoder
    Tools::Check(piEncoder->Initialize(piFileStream, WICBitmapEncoderNoCache));

    Tools::Check(piDecoder->GetFrameCount(&count));

    // Process each frame of the image.
    for (UINT i=0; i<count; i++)
    {
        // Frame variables.
        IWICBitmapFrameDecode *piFrameDecode = NULL;
        IWICBitmapFrameEncode *piFrameEncode = NULL;
        IWICMetadataQueryReader *piFrameQReader = NULL;
        IWICMetadataQueryWriter *piFrameQWriter = NULL;

        // Get and create the image frame.
        Tools::Check(piDecoder->GetFrame(
            i,
            &piFrameDecode)
            );

        Tools::Check(piEncoder->CreateNewFrame(
            &piFrameEncode,
            NULL)
            );

        // Initialize the encoder.
        Tools::Check(piFrameEncode->Initialize(
            NULL)
            );

        // Get and set the size.
        Tools::Check(piFrameDecode->GetSize(&width, &height));

        Tools::Check(piFrameEncode->SetSize(width, height));

        // Get and set the resolution.
        piFrameDecode->GetResolution(&dpiX, &dpiY);

        Tools::Check(piFrameEncode->SetResolution(dpiX, dpiY));

        // Set the pixel format.
        piFrameDecode->GetPixelFormat(&pixelFormat);

        Tools::Check(piFrameEncode->SetPixelFormat(&pixelFormat));

        // Check that the destination format and source formats are the same.
        bool formatsEqual = FALSE;
        GUID srcFormat;
        GUID destFormat;

        Tools::Check(piDecoder->GetContainerFormat(&srcFormat));
        Tools::Check(piEncoder->GetContainerFormat(&destFormat));

        if (srcFormat == destFormat)
            formatsEqual = true;
        else
            formatsEqual = false;

        if (formatsEqual)
        {
            // Copy metadata using metadata block reader/writer.
            piFrameDecode->QueryInterface(IID_PPV_ARGS(&piBlockReader));

            piFrameEncode->QueryInterface(IID_PPV_ARGS(&piBlockWriter));

            piBlockWriter->InitializeFromBlockReader(piBlockReader);
        }

        Tools::Check(piFrameEncode->GetMetadataQueryWriter(&piFrameQWriter));

        // Add additional metadata.
        /*PROPVARIANT padding;
        PropVariantInit(&padding);

        padding.vt = VT_UI4;
        padding.uintVal = 5012;

        Tools::Check(piFrameQWriter->SetMetadataByName(L"/xmp/PaddingSchema:Padding", &padding));
        Tools::Check(piFrameQWriter->SetMetadataByName(L"/app1/ifd/PaddingSchema:Padding", &padding));
        Tools::Check(piFrameQWriter->SetMetadataByName(L"/app1/ifd/exif/PaddingSchema:Padding", &padding));*/

        // Create XMP block
        IWICMetadataQueryWriter *pXMPriter = NULL;
        Tools::Check(piFactory->CreateQueryWriter(GUID_MetadataFormatXMP, NULL, &pXMPriter));

        PROPVARIANT xmpInfo;
        PropVariantInit(&xmpInfo);

        xmpInfo.vt = VT_UNKNOWN;
        xmpInfo.punkVal = pXMPriter;
        xmpInfo.punkVal->AddRef();

        piFrameQWriter->SetMetadataByName(L"/xmp", &xmpInfo);

        // Create XMP RegionInfos block
        IWICMetadataQueryWriter *pXMPRegionInfosWriter = NULL;
        Tools::Check(piFactory->CreateQueryWriter(GUID_MetadataFormatXMPStruct, NULL, &pXMPRegionInfosWriter));

        PROPVARIANT microsoftRegionInfo;
        PropVariantInit(&microsoftRegionInfo);

        microsoftRegionInfo.vt = VT_UNKNOWN;
        microsoftRegionInfo.punkVal = pXMPRegionInfosWriter;
        microsoftRegionInfo.punkVal->AddRef();

        piFrameQWriter->SetMetadataByName(L"/xmp/MP:RegionInfo", &microsoftRegionInfo);

        IWICMetadataQueryWriter *pXMPRegionWriter = NULL;
        Tools::Check(piFactory->CreateQueryWriter(GUID_MetadataFormatXMPBag, NULL, &pXMPRegionWriter));

        PROPVARIANT microsoftRegions;
        PropVariantInit(&microsoftRegions);

        microsoftRegions.vt = VT_UNKNOWN;
        microsoftRegions.punkVal = pXMPRegionWriter;
        microsoftRegions.punkVal->AddRef();

        piFrameQWriter->SetMetadataByName(L"/xmp/MP:RegionInfo/MPRI:Regions", &microsoftRegions);

        IWICMetadataQueryWriter *pXMPStructWriter = NULL;
        Tools::Check(piFactory->CreateQueryWriter(GUID_MetadataFormatXMPStruct, NULL, &pXMPStructWriter));

        PROPVARIANT xmpstruct;
        PropVariantInit(&xmpstruct);

        xmpstruct.vt = VT_UNKNOWN;
        xmpstruct.punkVal = pXMPStructWriter;
        xmpstruct.punkVal->AddRef();

        piFrameQWriter->SetMetadataByName(L"/xmp/MP:RegionInfo/MPRI:Regions/{ulong=0}", &xmpstruct);

        // Convert Plateform::String^ to LPWSTR
        std::vector<wchar_t> nameBuffer(peopleName->Begin(), peopleName->End());
        nameBuffer.push_back(0);

        PROPVARIANT valueName;
        PropVariantInit(&valueName);

        valueName.vt = VT_LPWSTR;
        valueName.pwszVal = nameBuffer.data();

        Tools::Check(piFrameQWriter->SetMetadataByName(L"/xmp/MP:RegionInfo/MPRI:Regions/{ulong=0}/MPReg:PersonDisplayName", &valueName));

        // Convert Plateform::String^ to LPWSTR
        std::vector<wchar_t> coordinatesBuffer(peopleCoordinates->Begin(), peopleCoordinates->End());
        coordinatesBuffer.push_back(0);

        PROPVARIANT valueCoordinates;
        PropVariantInit(&valueCoordinates);

        valueCoordinates.vt = VT_LPWSTR;
        valueCoordinates.pwszVal = coordinatesBuffer.data();

        Tools::Check(piFrameQWriter->SetMetadataByName(L"/xmp/MP:RegionInfo/MPRI:Regions/{ulong=0}/MPReg:Rectangle", &valueCoordinates));

        Tools::Check(piFrameEncode->WriteSource(
            static_cast<IWICBitmapSource*> (piFrameDecode),
            NULL)); // Using NULL enables JPEG loss-less encoding.

        // Commit the frame.
        Tools::Check(piFrameEncode->Commit());

        // Release
        if(piFrameQWriter)
        {
            piFrameQWriter->Release();
        }

        if(piFrameQReader)
        {
            piFrameQReader->Release();
        }

        if(piFrameEncode)
        {
            piFrameEncode->Release();
        }

        if(piFrameDecode)
        {
            piFrameDecode->Release();
        }

        if(piDecoder)
        {
            piDecoder->Release();
        }

        if(piFactory)
        {
            piFactory->Release();
        }
    }

    piEncoder->Commit();
    piFileStream->Commit(STGC_DEFAULT);
    if (piFileStream)
    {
        piFileStream->Release();
    }
    if (piEncoder)
    {
        piEncoder->Release();
    }
    if (piBlockWriter)
    {
        piBlockWriter->Release();
    }
    if (piBlockReader)
    {
        piBlockReader->Release();
    }
}

Code is well documented but basically, we start by creating a copy of the file, we enumerate all the sources properties and apply them to the new image and finally, we add our custom metadata. This last part is the most important because, in my mind, not enough documented.

Previously, you see a representation of the XMP metadata for people tagging so the goal here is to re-create the same structure (if this one has not been created). Otherwise, you just need to update it:

// Create XMP block
IWICMetadataQueryWriter *pXMPriter = NULL;
Tools::Check(piFactory->CreateQueryWriter(GUID_MetadataFormatXMP, NULL, &pXMPriter));

PROPVARIANT xmpInfo;
PropVariantInit(&xmpInfo);

xmpInfo.vt = VT_UNKNOWN;
xmpInfo.punkVal = pXMPriter;
xmpInfo.punkVal->AddRef();

piFrameQWriter->SetMetadataByName(L"/xmp", &xmpInfo);

// Create XMP RegionInfos block
IWICMetadataQueryWriter *pXMPRegionInfosWriter = NULL;
Tools::Check(piFactory->CreateQueryWriter(GUID_MetadataFormatXMPStruct, NULL, &pXMPRegionInfosWriter));

PROPVARIANT microsoftRegionInfo;
PropVariantInit(&microsoftRegionInfo);

microsoftRegionInfo.vt = VT_UNKNOWN;
microsoftRegionInfo.punkVal = pXMPRegionInfosWriter;
microsoftRegionInfo.punkVal->AddRef();

piFrameQWriter->SetMetadataByName(L"/xmp/MP:RegionInfo", &microsoftRegionInfo);

IWICMetadataQueryWriter *pXMPRegionWriter = NULL;
Tools::Check(piFactory->CreateQueryWriter(GUID_MetadataFormatXMPBag, NULL, &pXMPRegionWriter));

PROPVARIANT microsoftRegions;
PropVariantInit(&microsoftRegions);

microsoftRegions.vt = VT_UNKNOWN;
microsoftRegions.punkVal = pXMPRegionWriter;
microsoftRegions.punkVal->AddRef();

piFrameQWriter->SetMetadataByName(L"/xmp/MP:RegionInfo/MPRI:Regions", &microsoftRegions);

IWICMetadataQueryWriter *pXMPStructWriter = NULL;
Tools::Check(piFactory->CreateQueryWriter(GUID_MetadataFormatXMPStruct, NULL, &pXMPStructWriter));

PROPVARIANT xmpstruct;
PropVariantInit(&xmpstruct);

xmpstruct.vt = VT_UNKNOWN;
xmpstruct.punkVal = pXMPStructWriter;
xmpstruct.punkVal->AddRef();

piFrameQWriter->SetMetadataByName(L"/xmp/MP:RegionInfo/MPRI:Regions/{ulong=0}", &xmpstruct);

// Convert Plateform::String^ to LPWSTR
std::vector<wchar_t> nameBuffer(peopleName->Begin(), peopleName->End());
nameBuffer.push_back(0);

PROPVARIANT valueName;
PropVariantInit(&valueName);

valueName.vt = VT_LPWSTR;
valueName.pwszVal = nameBuffer.data();

Tools::Check(piFrameQWriter->SetMetadataByName(L"/xmp/MP:RegionInfo/MPRI:Regions/{ulong=0}/MPReg:PersonDisplayName", &valueName));

// Convert Plateform::String^ to LPWSTR
std::vector<wchar_t> coordinatesBuffer(peopleCoordinates->Begin(), peopleCoordinates->End());
coordinatesBuffer.push_back(0);

PROPVARIANT valueCoordinates;
PropVariantInit(&valueCoordinates);

valueCoordinates.vt = VT_LPWSTR;
valueCoordinates.pwszVal = coordinatesBuffer.data();

Tools::Check(piFrameQWriter->SetMetadataByName(L"/xmp/MP:RegionInfo/MPRI:Regions/{ulong=0}/MPReg:Rectangle", &valueCoordinates));

As you can see, we first start by created the XMPStruct (/xmp) then the XMPBag (/xmp/MP/RegionInfo). Then, another XMPStruct is created (/xmp/MP/RegionInfo/MPRI:Regions). Finally, for each person we want to tag, we add a region and put the name and rectangle of the (/xmp/MP/RegionInfo/MPRI:Regions/{ulong=0}/MPReg:PersonDisplayName).

Theses steps are very important because they allow you to create the basic metadata structure for people tagging. Without this, the code in the documentation won’t be able to work:

pQueryWriter->SetMetadataByName(
      L"/xmp/<xmpstruct>MP:RegionInfo/<xmpbag>MPRI:Regions/PersonDisplayName", &value);

In the previous code, we use the QureyWriter to set the metadata:

PROPVARIANT valueName;
PropVariantInit(&valueName);

valueName.vt = VT_LPWSTR;
valueName.pwszVal = nameBuffer.data();

Tools::Check(piFrameQWriter->SetMetadataByName(L"/xmp/MP:RegionInfo/MPRI:Regions/{ulong=0}/MPReg:PersonDisplayName", &valueName));

We set only the metadata for 1 people but you could easily modify the code to enumerate an array and use it.

Now that we’ve seen how to write metadata, we need to see how to read those metadata:

Platform::String^ TagMeManager::ReadPeopleNameFromFile(Platform::String^ filePath)
{
    IWICImagingFactory *piFactory = NULL;

    // WIC Factory
    Tools::Check(
        CoCreateInstance(
        CLSID_WICImagingFactory,
        nullptr,
        CLSCTX_INPROC_SERVER,
        IID_PPV_ARGS(&piFactory))
        );

    // Create WIC Decoder from file
    Tools::Check(piFactory->CreateDecoderFromFilename(
        filePath->Data(),
        NULL,
        GENERIC_READ,
        WICDecodeMetadataCacheOnDemand,
        &m_wicBitmapDecoder)
        );

    // Get frame from decoder
    Tools::Check(m_wicBitmapDecoder->GetFrame(
        0,  //JPEG has only one frame.
        &m_wicBitmapFrameDecode)
        );

    // Get metadata query reader
    Tools::Check(m_wicBitmapFrameDecode->GetMetadataQueryReader(
        &m_wicMetadataQueryReader)
        );

    Platform::String^ peopleName;

    PROPVARIANT personNameValue;
    PropVariantInit(&personNameValue);

    try
    {
        // Put code in try/catch because the property might not exist if the image is not tagged

        /*Tools::Check(m_wicMetadataQueryReader->GetMetadataByName(L"System.Photo.PeopleNames", &personNameValue));*/
        Tools::Check(m_wicMetadataQueryReader->GetMetadataByName(L"/xmp/MP:RegionInfo/MPRI:Regions/{ulong=0}/MPReg:PersonDisplayName", &personNameValue));

        peopleName = ref new Platform::String(personNameValue.pwszVal);
    }
    catch(Exception^ e)
    {
        // Do nothing
    }

    if(piFactory)
    {
        piFactory->Release();
    }

    return peopleName;
}

Platform::String^ TagMeManager::ReadPeopleRectangleFromFile(Platform::String^ filePath)
{
    IWICImagingFactory *piFactory = NULL;

    // WIC Factory
    Tools::Check(
        CoCreateInstance(
        CLSID_WICImagingFactory,
        nullptr,
        CLSCTX_INPROC_SERVER,
        IID_PPV_ARGS(&piFactory))
        );

    // Create WIC Decoder from file
    Tools::Check(piFactory->CreateDecoderFromFilename(
        filePath->Data(),
        NULL,
        GENERIC_READ,
        WICDecodeMetadataCacheOnDemand,
        &m_wicBitmapDecoder)
        );

    // Get frame from decoder
    Tools::Check(m_wicBitmapDecoder->GetFrame(
        0,  //JPEG has only one frame.
        &m_wicBitmapFrameDecode)
        );

    // Get metadata query reader
    Tools::Check(m_wicBitmapFrameDecode->GetMetadataQueryReader(
        &m_wicMetadataQueryReader)
        );

    Platform::String^ rectangle;

    PROPVARIANT rectangleValue;
    PropVariantInit(&rectangleValue);

    try
    {
        // Put code in try/catch because the property might not exist if the image is not tagged
        Tools::Check(m_wicMetadataQueryReader->GetMetadataByName(L"/xmp/MP:RegionInfo/MPRI:Regions/{ulong=0}/MPReg:Rectangle", &rectangleValue));

        rectangle = ref new Platform::String(rectangleValue.pwszVal);
    }
    catch(Exception^ e)
    {
        // Do nothing
    }

    if(piFactory)
    {
        piFactory->Release();
    }

    return rectangle;
}

Here again, the useful part of the code is the use of the QueryReader to get the metadata:

PROPVARIANT personNameValue;
PropVariantInit(&personNameValue);

try
    {
        // Put code in try/catch because the property might not exist if the image is not tagged
        Tools::Check(m_wicMetadataQueryReader->GetMetadataByName(L"/xmp/MP:RegionInfo/MPRI:Regions/{ulong=0}/MPReg:PersonDisplayName", &personNameValue));

        peopleName = ref new Platform::String(personNameValue.pwszVal);
    }
    catch(Exception^ e)
    {
        // Do nothing
    }

This code could be arrange to get the name and rectangle in one method (and also to get all the names/rectangles) but it gives you the first step to understand how it works.

Demonstration

Ok, our component for reading/writing people metadata is ready so let’s take a look at how to use it. Hopefully, thanks to WinRT projections, it’s really easy!

Here is a simple demo app that allows you to load an image.

If the image has people tagging metadata, we display the information on screen:

Screenshot (75)

If no metadata are set, we give users the possibility to define:

  1. The rectangle containing the person to tag
  2. The name of this person

Screenshot (77)

The C# code is really easy because the most complicated part is done in C++. Indeed, to read the metadata, we just need to use:

this.tbName.Text = _tagHelper.ReadPeopleNameFromFile(this._tempImage.Path);
this.tbCoordinates.Text = _tagHelper.ReadPeopleRectangleFromFile(this._tempImage.Path);

Similarly, the code to set the metadata is also very simple:

var borderLeft = Canvas.GetLeft(this.tagger) / this.preview.ActualWidth;
var borderTop = Canvas.GetTop(this.tagger) / this.preview.ActualHeight;
var borderWidth = this.tagger.ActualWidth / this.preview.ActualWidth;
var borderHeight = this.tagger.ActualHeight / this.preview.ActualHeight;

var coordinates = string.Format("{0},{1},{2},{3}", borderLeft, borderTop, borderWidth, borderHeight);

var copy = await this.GetCopyImage(this._tempImage);

this._tagHelper.AddPeopleNameToFile(this._tempImage.Path, copy, user, coordinates);

 

So as you can see, people tagging feature can be really cool and simple to implement in your app. Of course, the previous code is a simple prototype so if you want to grab it and improve it, feel free and let me know !

You can download the source code here.

 

Happy coding!

You can follow any responses to this entry through the RSS 2.0 You can leave a response, or trackback.

2 Responses

Add Comment Register



Leave a Reply