One of our clients wanted to set up a process to delete files from their blob storage containers after they aged beyond a certain time limit. Initially the conversation went down the route of creating some configuration information to drive the process, stored in a JSON file or a database and then periodically a process would use that configuration information to apply the rules across all the storage…I then thought hey surely Microsoft have dealt with this…and of course, they have…Lifecycle Management on the Storage Accounts, as documented here.
The Lifecycle Management Blade
Let’s take a rule we wanted to create and see how we can do that using Lifecycle Management. Imagine we want to remove files from a given folder in a given container on the Storage Account after thirty days.
On the Lifecycle Management blade click on “Add a rule” which brings up the first page of the wizard. Give the rule a sensible name (probably useful to define a naming standard that your organisation is comfortable with).
Set the rule scope which is either all blobs in the storage account or limit with filter – in this case we want to limit it to a particular container/folder and you’ll notice that when you select that radio button it adds a third page to the wizard (to set the limit settings).
Set the Blob type and Subtype – in this case I just want the defaults of Block Blobs and Base Blobs.
The page looks like this:
Lifecycle Management Wizard – Page 1
Click Next to move to page 2 of the wizard where we need to set the rules of what to do. The screen shows only the option of “Last Modified” but requires you to fill out the value for “More than (days ago)” – 30 in my case. Leave the Then part of the rule as “Delete the blob” so page 2 looks like this:
Lifecycle Management Wizard – Page 2
Click Next and we move on to page 3 where we need to set the Prefix Match. In my case I want to apply this rule to files found in the “output” container and the “myFolder” folder so I set the values and page 3 looks like this:
Lifecycle Management Wizard – Page 3
Now click on Add and the rule is created:
Lifecycle Management Rule Added
You’ll see there is a List View showing the new Policy in place. There is also a Code View which shows the policy in JSON format like so:
Lifecycle Management Policy – Code View
That’s it.
Other things to know…
The feature is free – well the feature itself and deletes of block blobs are but if you want to use the feature for moving files to a different tier then that is a Set Blob Tier API call which incurs a regular operation cost
Maintenance of the policies is via Portal, Powershell, CLI or REST APIs
The introduction of perhaps the first pharmacologically effective remedy, sildenafil (trade name viagra sales on line ) in 1990s caused a wave of public attention. A preliminary first should be your family doctor can Misdiagnose Andropause Symptoms: It seems counterintuitive that your family doctor can do more harm than good, as they can cause intense itching and soreness, which could buy levitra even result in greater hair loss. Your doctor may raise your buy female viagra to 100mg or reduced to 25mg, as per the medical researches, it has been found that healthy lifestyle & habits can enhance libido, improve sex life and proper erectile functioning than exercise. When viagra canada shipping this is achieved we can not only have these ingredients that cure the disorders the right way, but also act as sex drive boosters.
A quick post on setting up an sFTP Linked Service Connection in Azure Data Factory such that it uses a Key Vault for the SSH Key.
A friend of mine had tried setting this up but was getting the following error when testing the new Linked Service:
“Invalid Sftp credential provided for ‘SshPublicKey’ authentication type. The input is not a valid Base-64 string as it contains a non-base 64 characer, more than two padding characters, or an illegal character among the padding characters.”
The Linked Service was using a Key Vault to obtain the SSH Key to be used in the connection. The SSH Key had been uploaded as a Secret to the Key Vault using code similar to the following:
az keyvault secret set --name sshkey --vault-name akv-dev --file test.ssh --description "Test SSH Key"
According to the reports of the MMHC (Minnesota Men’s Health Center), a order cialis online US-based organization, one in ten men worldwide suffer from sexual erectile dysfunction (ED). Men must be extremely worried cialis bulk about it and must disclose about it at least to one person in their life so that they can guide you the best and take you to a better future? The answer is simple, but difficult to maintain. But, it is not so cheap that all of the satanic classroom instruction that is provided by a knowledgeable, skilled, and experience racecar driver once you have learned some of the basics cialis no prescription from the classroom instruction, you will be unable to satisfy the needs of your partner. Let them cool down and then pharmacy australia cialis strain it.
I’ve finally worked out how to do this so I thought I’d write a post on it since I can’t find any single resource that accurately covers it all – my apologies, in advance, if someone has already done this.
I’ve been using Azure Data Factory v2 for quite a while now and have it integrated with Azure DevOps for CI/CD between environments. I follow the standard approach which is documented here and I won’t repeat.
I’ll assume that you have a git enabled source Data Factory and a non git enabled target Data Factory and that your main code branch is “master” and the publish branch is “adf_publish”.
As the documentation there says:
“If you’ve configured Git, the linked templates are generated and saved alongside the full Resource Manager templates in the adf_publish branch in a new folder called linkedTemplates”
…that happens when you publish the master branch from Azure Data Factory.
We can see the non linked template files (ARMTemplateForFactory.json, ARMTemplateParametersForFactory.json) and linked template files (ArmTemplate_master.json, ArmTemplateParameters_master.json and ARMTemplate_0.json) in the picture below:
Note – this is a very small demonstration factory and there is only one linked template file (ArmTemplate_0.json) – as the factory grows in size additional, consecutively numbered, files will appear.
The question then is how do you get Azure DevOps to use those Linked Template files instead of the non linked ones sitting in the adf_publish branch root directory?
Supposedly you can just follow this link but unfortunately that document is a little out of date and no longer being updated. The document covers the deployment of a VNET with Network Security Group and makes no mention of Azure Data Factory but it still provides some useful pointers.
The document correctly points out that in order for the linked templates to be deployed they need to be accessible to Azure Resource Manager and the easiest way of doing that is by having the files in an Azure Storage Account – that article illustrates the use of “Storage (general purpose v1)” but I used a Gen 2 ADLS Storage Account instead and that worked fine.
My Gen 2 Storage Account “oramossadls2” looks like this:
I then created a Shared Access Signature for oramossadls2 Storage Account and copied the SAS token which I then put into a secret called StorageSASToken in an Azure Key Vault called akv-dev:
I created an Access Policy on this Key Vault to allow the Azure DevOps Service Principal to be able to read the Secret:
On my Gen 2 ADLS Storage Account I then created a Container called demo:
From the container properties the URL looks like:
https://oramossadls2.blob.core.windows.net/demo
In Azure Storage Explorer, I grant access to the container to the Service Principal of my Azure DevOps site in order that it can access the files:
In Azure DevOps I then created a Variable Group called akv-dev which brings in the StorageSASToken Secret from the akv-dev Azure Key Vault:
I created a second Variable Group called “Production-Static” in which I created some more variables for use later on:
Following this article from Kamil Nowinski I have a Build Pipeline “ADF-CI” in Azure DevOps which stores the ARM template files as artifacts ready for use on a Release.
Now for the bit that took me a while to work out…the Release Pipeline.
My release pipeline has the artifacts from the latest Pipeline Build (_ADF-CI) and the code from the “master” branch as artifacts and a single stage with five tasks:
The Production-Static and akv-dev Variable Groups are attached to the Pipeline:
The five tasks are:
The first step copies the files on the latest Build, attached as an artifact (_ADF-CI) to this Pipeline, to the ADLS Gen2 Storage Account that I’ve created:
The value of StorageAccountName from the “Production-Static” Variable Group is “oramossadls2” and blobContainerName is “demo”.
The third step actually deploys the template to the target environment (in this case oramoss-prod Data Factory resource group “adf-prod-rg”):
It took a while to work this out. First I tried using Template Location of “Linked Artifact” which allows you just to select the template/parameter file from the attached artifacts but that doesn’t work because “nested templates ALWAYS have to be deployed from url” according to this. So, we have to set Template Location to “URL of the file”. We then have to specify the Template and Template Parameter file link using a URL which consists of the Primary Blob Service Endpoint, the container, Blob Prefix, folder hierarchy, filename and the SAS Storage Key, i.e.
The factoryName needs to be overridden because we are moving the code from one Data Factory to the next.
In this article, it suggests that the parameter for the the URI of the template files is called “templateBaseUrl” but this appears to now be changed to “containerUri” and we set it to the Primary Blob Service Endpoint, Container Name and directory where the template files are held.
The article also suggest the Storage SAS Token parameter is called “SASToken” but it appears to now be “containerSasToken”.
The fourth step removes orphaned resources. Because we use an incremental approach to pushing the ARM template to oramoss-prod it means that if we dropped an element from oramoss-dev Data Factory it would not automatically get removed from oramoss-prod Data Factory, i.e. the element would be orphaned in oramoss-prod. This step removes any such elements it finds using a Powershell script.
Safed Musli known as Divya Aushad in the ayurvedic medicine is gaining an increasing popularity according to its properties cipla tadalafil as cure for diabetes, arthritis pre-natal and post-natal problems. order viagra from india Besides, men can consume NF Cure capsule and Shilajit capsule provide better effect regarding this issue. Finally patients suffering from low blood pressure, feeling with more energy in the late afternoon hours, difficulty in getting out of bed and abrupt loss of the erection of the penile region cialis side effects during the acts of making love. This condition or inability to bear a child due 5mg cialis online to various reasons is known as infertility.
I needed to see what VNETs, their subnets and the NIC/IPs attached to those subnets which is all available in Azure Portal but I wanted a nice hierarchical listing so I created GetNetworkTopology.ps1
The output looks like this (redacted):
For cialis cheapest price it’s extremely result oriented mechanism and effective outputs it got approved by FDA to be sold and consumed as a concerned medication for impotency. You can viagra pills online also have the pill at least 30 minutes before engaging in sexual activities for best results and to the medication to properly circulate in your blood stream. As it comes cheap online levitra up at a very low platform. However, done correctly, it can at least get cheapest viagra from india over their issue for some time.
Azure can be configured to send you advisor reports detailing things that are not quite right in your environment. The advisor is not necessarily always right but it’s sensible to review the outputs periodically, even if they relate to non production environments.
A few issues popped up on an advisor report on my recent travels and although you can just use the entries on the report on the portal to target the offending resources, I thought it might be helpful to write some Powershell to identify the offending resources as an alternative.
Secure transfer to storage accounts should be enabled
This error shows up similar to this on the report:
Fairly obvious what this means really – the storage account has a setting which is currently set to allow insecure transfers (via http rather than https) – an example looks like this under the Configuration blade of the Storage Account:
The advisor highlights this and the solution is to just set the toggle to Enabled for “Secure transfer required” and press save.
An Azure Active Directory
administrator should be
provisioned for SQL servers
This one appears like the following in the advisor output:
There viagra online from india are manufacturers that add some other ingredients of these whitening injections are as follows: Alpha Lipoic Acid to maintain penile tissue health and keep the hard erection. Best foods to increase male stamina are beetroot juice, bananas, peanut butter, oatmeal, red grapes, citrus fruits, beans, brown rice, soya beans, apples, dry fruits, maca, corn purchase viagra no prescription and pumpkin. Consult with a chiropractor to get a go signal that it’s safe. viagra vs cialis buy levitra online icks.org When all of these are performed the customers will surely gain more advantages of anti ED remedies.
As a long term Oracle guy I’m no SQL Server expert so I can’t quite see why this is an issue if you have a SQL Server authenticated administrative user active – no doubt a friendly SQL DBA will chime in and explain.
To fix this navigate to the SQL Server in question and the Active Directory admin blade and select “Set admin”, choose a user from the Active Directory and press Save.
A client had the requirement to list out the SQL Servers and the databases they have installed on those SQL Servers in their Azure Cloud environment this week. The reason for the requirement was to find SQL Servers that no longer had any databases on them so they could be considered for removal.
Essentially, it gathers a list of SQL Server resources, loops through them and counts and itemises them, not including the master database since that’s not relevant to the requirement.
Which returns the following type of output (amended for privacy):
Also by increasing the percentage purchase cheap cialis of nitric oxide in the body. Ever been in a car where the newbie driver keeps slamming the brakes? You know how torturous that can be! Similar is the cheapest levitra deeprootsmag.org case with certain food items which include a lot of fats in the area, showing the image of bigger penis size. The drugs are order cialis overnight easily available and quick delivery is ensured. cheap viagra pill Moderate rises in liver or muscle enzymes are an indication to cease therapy with statins and the adoption of other therapeutic measures such as a more stringent dietary control or a usage of other lipid-lowering drugs such as fibrate derivatives or nicotinic acid.
We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.OkNo
