GeezerVison is showing github.com — Simplify usage of Multipart Uploads with WebRequestPSCmdlet · Issue #2112 · PowerShell/PowerShell
Simplify usage of Multipart Uploads with WebRequestPSCmdlet #2112DescriptionReacted by Keith Hill, Florian Feldhaus, Andreas Hunkeler and Joshua Todd Cowper Activityswinster commented on Aug 22, 2017 iSazonov commented on Aug 23, 2017 markekraus commented on Aug 25, 2017 Ilya (@iSazonov) sure. I will take a look at it. I have seen other requests in the past few years for multipart forms in general as several IoT APIs seem to only work with multipart forms. iSazonov commented on Aug 25, 2017 Feel free to ping MSFT experts here if there is compatibility issues. swinster commented on Aug 25, 2017 FWIW, we mainly Python, Django and Apache for our Web interfaace stuff. This is all RESTfull APIs. I am not developer, just attempting to interact with the API as we are seeing more sysadmins wanting to do this. markekraus commented on Aug 25, 2017 swinster if you could provide me an exact scenario that would be very useful. Right now, all I have to go off of is to build a out a test API that takes multipart forms and files and hope that fits most scenarios. swinster commented on Aug 25, 2017 Mark Kraus (@markekraus) sure. I work for Pexip, and the entire platform a can be automated via API calls. This specific example is required with regard to uploading an upgrade file to the platform, which will be in excess of 1GB. The API documentation can be found here - https://docs.pexip.com/api_manage/api_configuration.htm?Highlight=upgrade
I can even post here the resultant function created to achieve this, although this is based on the information post by Mario Majčica in his blog above. markekraus commented on Aug 25, 2017 swinster awesome. If I need more I'll reach out. markekraus commented on Aug 25, 2017 swinster Can you share your function with me? i want to make sure I'm on the right path. markekraus commented on Aug 26, 2017 This presents some interesting implementation issues. To truly support Multipart form data, the Cmdlets would need to accept multiple files, a field name for each file, possibly a content type for each file, and the ability to accept a dictionary of field/value pairs at the same time. Additionally, It appears some APIs are particular about the boundary used, so an optional boundary would need to be supplied. Attempting to make that mesh with the current Cmdlets would be a significant rework almost to the point of justifying separate cmdlets for Multipart support.
The issue is in Multipart form data being extremely flexible and radically different from other requests. It would be easy to tack on support for a single file or to convert a -Body dictionary, but it would not be simple to mix a file with form data or to support multiple files. Fitting all the Multipart use-cases into the current cmdlets would touch a large portion of code to accommodate a valid but somewhat less common usage of the cmdlets.
IMO, It would also lead to a somewhat confusing UX. If -InFile was used for Multipart files and only accepted multiple files on a Multipart request it would lead to issues with users supplying multiple paths on a standard request, getting errors, and wondering why it accepts an array of paths. Also, explaining that -Body and -InFile are mutually exclusive except when doing Multipart requests would be somewhat confusing confusing. If files were split into a separate and new parameter like -MultipartFile, that might cause confusion as to why -InFile doesn't work on Multipart requests or why there needs to be a difference in the first place.
The problem I see with simplifying this is that many of the requests I've read for sending Multipart form data through PowerShell have only their own use-case in mind. It might seem simple from that perspective, but when you look a dozen or so of these requests you begin to see they almost all are distinctly different needs falling under the broad category of Multipart form data. A Narrow implementation would leave many use-cases in their current state, a broad implementation would impact a lot of code but for feature addition that is less commonly used.
With that in mind, I think a fair compromise and better approach would be to accept a -Body object/collection that can be used by the Cmdlets to create the Multipart Form-Data Request. This could be simplified by exposing new cmdlet(s) and new classes to simplify generating that object/collection. Body could simply be adapted to accept MultipartFormDataContent at least easing the burden of the user managing an HttpClient.
I could start by adding support for MultipartFormDataContent in -Body. From there it could be decided to add convenience cmdlets or possibly even wrapper classes to easy creation.
James Truher (@JamesWTruher) Dan Travison (@dantraMSFT) Paul Higinbotham (@PaulHigin) Steve Lee (@SteveL-MSFT) What are your thoughts? swinster commented on Aug 26, 2017 Mark Kraus (@markekraus) FWIW, here is the modified function from Mario's blog.
The main addition was the Basic Authentication headers which also required a Base64 encoded version of the authentication credentials. Initially, I decrypted the PSCredential password via a separate function, however this was changed to a single line. The Basic Authroziation header is required as although you can provide simple credentials to the the HTTPClient (and indeed the Invoke-WebRequest), the first request is sent without this header, which is then challenged, thus a second request is made. This means that then entire 1 GB file is uploaded twice - a huge waste of time and resources.
As an aside, I have a similar issue (WRT the Basic Authorisation Header) when downloading a large file (3-4 GB), again again leads to a two request issue, and the download of the file is actioned twice), However the Invoke-WebRequest is even worse in that case as the entire file is streamed into memory and cause the remote server to lock up when I tried to use this CmdLet. I eventually changed to use the WebClient (I couldn't get the HTTPClient to work with my lacking programming knowledge) , however, WebClient doesn't implement a timeout property so you have to build a class to inherit and extend the WebClient. All very, very complex and confusing, especially to someone like me with little programming expertise. As mentioned, this is an aside to this particular issue, but is similar in nature.
function Invoke-MultipartFormDataUpload
{
[CmdletBinding()]
PARAM
(
[string][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$InFile,
[string]$ContentType,
[Uri][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$Uri,
[System.Management.Automation.PSCredential]$Credential
)
BEGIN
{
if (-not (Test-Path $InFile))
{
$errorMessage = ("File {0} missing or unable to read." -f $InFile)
$exception = New-Object System.Exception $errorMessage
$errorRecord = New-Object System.Management.Automation.ErrorRecord $exception, 'MultipartFormDataUpload', ([System.Management.Automation.ErrorCategory]::InvalidArgument), $InFile
$PSCmdlet.ThrowTerminatingError($errorRecord)
}
if (-not $ContentType)
{
Add-Type -AssemblyName System.Web
$mimeType = [System.Web.MimeMapping]::GetMimeMapping($InFile)
if ($mimeType)
{
$ContentType = $mimeType
}
else
{
$ContentType = "application/octet-stream"
}
}
}
PROCESS
{
Add-Type -AssemblyName System.Net.Http
$httpClientHandler = New-Object System.Net.Http.HttpClientHandler
if ($Credential)
{
$networkCredential = New-Object System.Net.NetworkCredential @($Credential.UserName, $Credential.Password)
$httpClientHandler.Credentials = $networkCredential
$httpClientHandler.PreAuthenticate = $true
$httpClient = New-Object System.Net.Http.Httpclient $httpClientHandler
#$password = Get-PlainText -SecureString $Credential.Password
$Base64Auth = [System.Convert]::ToBase64String([System.Text.Encoding]::GetEncoding("iso-8859-1").GetBytes([String]::Format( "{0}:{1}", $Credential.UserName, $Credential.GetNetworkCredential().Password)))
#$Base64Auth = [Convert]::ToBase64String([Text.Encoding]::GetEncoding("iso-8859-1").Getbytes("$($Credential.UserName):$($password)"))
$httpClient.DefaultRequestHeaders.Add("Authorization", "Basic $Base64Auth")
}
else {
$httpClient = New-Object System.Net.Http.Httpclient $httpClientHandler
}
$httpClient.Timeout = 18000000000
#$httpClient.DefaultRequestHeaders.Add("AUTHORIZATION", "Basic YTph")
$packageFileStream = New-Object System.IO.FileStream @($InFile, [System.IO.FileMode]::Open)
$contentDispositionHeaderValue = New-Object System.Net.Http.Headers.ContentDispositionHeaderValue "form-data"
$contentDispositionHeaderValue.Name = "package"
$contentDispositionHeaderValue.FileName = (Split-Path $InFile -leaf)
$streamContent = New-Object System.Net.Http.StreamContent $packageFileStream
$streamContent.Headers.ContentDisposition = $contentDispositionHeaderValue
$streamContent.Headers.ContentType = New-Object System.Net.Http.Headers.MediaTypeHeaderValue $ContentType
$content = New-Object System.Net.Http.MultipartFormDataContent
$content.Add($streamContent)
try
{
$response = $httpClient.PostAsync($Uri, $content).Result
if (!$response.IsSuccessStatusCode)
{
$responseBody = $response.Content.ReadAsStringAsync().Result
$errorMessage = "Status code {0}. Reason {1}. Server reported the following message: {2}." -f $response.StatusCode, $response.ReasonPhrase, $responseBody
throw [System.Net.Http.HttpRequestException] $errorMessage
}
#return $response.Content.ReadAsStringAsync().Result
return $response
}
catch [Exception]
{
$PSCmdlet.ThrowTerminatingError($_)
return $response
}
finally
{
if($null -ne $httpClient)
{
$httpClient.Dispose()
}
if($null -ne $response)
{
$response.Dispose()
}
}
}
END { }
}
swinster commented on Aug 26, 2017 I have also hardcoded the ContentDispositionHeaderValue value (package) which is all that was required for my needs. lzybkr commented on Dec 7, 2017 if [file] -eq [system.io.fileinfo], I think that will cause confusion, e.g.
using namespace System.IO
[File]::WriteAllLines($path, $line) Clearly that's not meant to be FileInfo, but if you don't see the using, you might be confused if you get used to see [File] in other places. markekraus commented on Dec 7, 2017 For me, the name isn't so important as just having easy accelerators for files and directories. I'm not a huge fan of [system.io.fileinfo] not throwing on the file not existing. ideally this accelerator would be used like this
function Get-Widget {
param([PSFile]$File)
# Do something with $File
}
Get-Widget -File 'x:\no\lo\existo.nope'
# parameter binding error for non existent file
Get-Widget -File 'c:\some\real\file.txt'
# works with existing file
$form = @{
StringField = 'stringValue'
FileField = [PSFile]'x:\no\lo\existo.nope'
}
# type conversion error on non existing file
$form = @{
StringField = 'stringValue'
FileField = [PSFile]'c:\some\real\file.txt'
}
# no errormarkekraus commented on Dec 7, 2017 also... I'd like to note that the accelerators should not be blocking to the multipart implementation. I can implement this now with out [file] and add support for that later. The accelerators should have been tied to a separate issue. while having them would make the UX for the -Form feature easier, for now the user can do
$form = @{
FileField = Get-Item 'c:\some\real\file.txt'
}
iwr -Form $form -Method POST -Uri $uriSteveL-MSFT commented on Dec 7, 2017 The accelerators should be a separate issue and PR. You can still use [System.IO.FileInfo] for now. swinster commented on Dec 28, 2017 Just a quick question. with regard to this issue, has anything made it to PowerShell v6 as yet ?
Not sure if the underlying mechanism to retrieve a file will be altered, but it came to light the other day that the way I have the "working" at the moment in v5 is a little on the slow side. An HTTP download of a 4GB file from a browser from a server based on the same network takes approximately 30 seconds. In PowerShell v5, this takes around 1 hour !!! markekraus commented on Dec 28, 2017 swinster commented on Dec 28, 2017 markekraus commented on Dec 28, 2017
Can't wait to be able to retrieve files with -outfile
swinster Can you elaborate on this? You can already download files and use the -OutFile parameter to save them. This has been around for several versions at least.
The blog post I linked has an earlier version of what was planned for the simplified multipart/form-data support. The planned implimentation has since been revised. You can see a "demo" of the currently planned simplified multipart/form-data support here https://get-powershellblog.blogspot.com/2017/12/powershell-core-web-cmdlets-in-depth_24.html#L21 and you can see the proposal here #2112 (comment) swinster commented on Dec 28, 2017 Mark Kraus (@markekraus) apologies, the comment above was not in the correct place as it related to the underlying method of data retrieval from a simple GET method on the Invoke-WebRequest, which was in the order of 200 time slower that a browser request, but actually nothing to do the MultiPart uploads. However, I have just run the same basic tests using Invoke-WebRequest to GET a large file in v6 and v5 simultaneously and honestly there is a HUGE difference - whoever made these changes to the implementation here need some applause as well - even if this is the wrong place! markekraus commented on Jan 20, 2018 An update for anyone following this issue. I am currently working on implementing the solution as approved by the PowerShell Committee with some minor adjustments. I hope to have a PR submitted in the coming days.
After reading RFC-7578 it became clear that a single field name can be supplied multiple times with different field values. The use case for this is an array of files (multiple file selection) or values for a single form field. To accommodate that, I'm adjusting the implementation slightly to treat collections as multiple field values for the same field name.
for example:
$Form = @{
Files = Get-ChildItem c:\temp -File
Names = "Bill", "Sue", "Jane"
Cars = [System.Collections.Generic.List[String]]::new([String[]]@("Smart", "Honda"))
Description = "Multiple and single value support."
Image = [System.IO.FileInfo]"c:\picture\me.png"
}
Invoke-WebRequest -Method POST -Uri $uri -Form $FormIn the approved implementation, this would result in a collection being converted to a single string. In the implementation I am working on, Names would be supplied 3 times with Bill as the first value, Sue as the second, and Jane as the third. It doesn't add much complexity and initial testing shows it works for endpoints that support it. It provides more flexibility at minimal cost. After checking, I personally have an internal use case for this in a form that requires multiple values for a single field.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|