{"id":724,"date":"2019-10-03T11:55:20","date_gmt":"2019-10-03T11:55:20","guid":{"rendered":"http:\/\/172.23.1.43\/?p=724"},"modified":"2022-06-07T22:25:59","modified_gmt":"2022-06-07T22:25:59","slug":"powershell-utf8-and-bom","status":"publish","type":"post","link":"https:\/\/blog.n-dol.org\/2019\/10\/03\/powershell-utf8-and-bom\/","title":{"rendered":"PowerShell &#8211; UTF8 and BOM"},"content":{"rendered":"\n<p>When you save something  as UTF-8 in PowerShell, it, most likely, will be as UTF-8 with BOM.  This can be a problem if there is characters not in the<a rel=\"noreferrer noopener\" aria-label=\" 7-bit ASCII character set (opens in a new tab)\" href=\"https:\/\/en.wikipedia.org\/wiki\/ASCII\" target=\"_blank\"> 7-bit ASCII character set<\/a> it won&#8217;t be handled properly. So latin, non-latin, chinese, &#8230; are not handled by it.<\/p>\n\n\n\n<p>The good news is from PowerShell 6+ the encoding is UTF8.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What is BOM ?<\/h2>\n\n\n\n<p>BOM stand for Byte Order Mark. It is an optional Unicode character. It indicate the <a rel=\"noreferrer noopener\" aria-label=\"endianness (opens in a new tab)\" href=\"https:\/\/en.wikipedia.org\/wiki\/Endianness\" target=\"_blank\">endianness<\/a> of the encoding.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How to save in UTF-8 without BOM<\/h2>\n\n\n\n<p>In a nutshell what you need to set false on the attribute with:<\/p>\n\n\n\n<pre class=\"wp-block-prismatic-blocks\"><code class=\"language-csharp\">New-Object System.Text.UTF8Encoding($false)<\/code><\/pre>\n\n\n\n<p>After that, it&#8217;s a matter to save the encoding setting for the file. Here&#8217;s 2 examples XML and text file as a bonus the text file example manipulate JSON file.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">XML Edition<\/h3>\n\n\n\n<pre class=\"wp-block-prismatic-blocks\"><code class=\"language-powershell\">[XML] $XmlDocument = ( Select-Xml -Path $FilePath  -XPath \/ ).Node\n\n[System.Xml.XmlWriterSettings] $XmlSettings = New-Object System.Xml.XmlWriterSettings\n\n#Preserve Windows formating\n$XmlSettings.Indent = $true\n\n#Keeping UTF-8 without BOM\n$XmlSettings.Encoding = New-Object System.Text.UTF8Encoding($false)\n\n[System.Xml.XmlWriter] $XmlWriter = [System.Xml.XmlWriter]::Create($FilePath, $XmlSettings)\n$XmlDocument.Save($XmlWriter)\n\n#Close Handle and flush\n$XmlWriter.Dispose()<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Text file Edition<\/h3>\n\n\n\n<p>Here&#8217;s is a JSON example.<\/p>\n\n\n\n<pre class=\"wp-block-prismatic-blocks\"><code class=\"language-powershell\">$FilePath = &quot;C:\\MyFolder\\textfile.json&quot;\n$FileContent = Get-Content -Path $FilePath | ConvertFrom-Json\n\n$UTF8Only = New-Object System.Text.UTF8Encoding($false)\n[System.IO.File]::WriteAllLines($FilePath, @($FileContent | ConvertTo-Json), $UTF8Only)\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>When you save something as UTF-8 in PowerShell, it, most likely, will be as UTF-8 with BOM. This can be&hellip;<\/p>\n","protected":false},"author":5614970,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[178495,672890843],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.n-dol.org\/wp-json\/wp\/v2\/posts\/724"}],"collection":[{"href":"https:\/\/blog.n-dol.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.n-dol.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.n-dol.org\/wp-json\/wp\/v2\/users\/5614970"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.n-dol.org\/wp-json\/wp\/v2\/comments?post=724"}],"version-history":[{"count":20,"href":"https:\/\/blog.n-dol.org\/wp-json\/wp\/v2\/posts\/724\/revisions"}],"predecessor-version":[{"id":1411,"href":"https:\/\/blog.n-dol.org\/wp-json\/wp\/v2\/posts\/724\/revisions\/1411"}],"wp:attachment":[{"href":"https:\/\/blog.n-dol.org\/wp-json\/wp\/v2\/media?parent=724"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.n-dol.org\/wp-json\/wp\/v2\/categories?post=724"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.n-dol.org\/wp-json\/wp\/v2\/tags?post=724"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}