Repository navigation
UTF-8, MIME Type Declaration, BOMs, and other encoding standards #198
Description
This is going to be a very general issue ticket in order to discuss and address a few inconsistencies with node projects, including ours. And I'd like to hear some experiences encountered with our node project and any others.
DONE - UTF-8: All node projects should be using uppercase UTF-8 and not utf-8 according to the spec. When this is not consistent this can lead to unpredictable behavior between deployments whether on dev or production servers. Some book page websites may say it can be lowercase but some software components may not be smart enough to distinguish the difference.
DONE - Byte Order Mark (BOM) : BOMs are said to be not used everywhere... currently we have one in at least one file. This should be remedied on a system that can control that. This from my experience usually happens when there is a Unicode character inserted into a file and saved. Our current STYLEGUIDE hints at this with "Avoid use of international characters
because they may not read well or be understood everywhere.". Unfortunately I don't see an easier way to detect if a pr or commit is generating these or not.
In general encodings may need to be explicitly defined in contradiction to our current STYLEGUIDE saying the server handles it.
_EDIT_:
MIME types: These should always be included rather than having the server/client guessing off of file extensions.
See also:
- http://tools.ietf.org/html/rfc3629#section-4
- http://stackoverflow.com/questions/4990095/json-specification-and-usage-of-bom-charset-encoding
- Clean up #173 (comment) a BOM noticed here.
Applies to and isolated from #19. Most of this will go in either STYLEGUIDE and/or CONTRIBUTING
Activity
sizzlemctwizzle commentedon Jun 24, 2014
Moved the UTF-8 bug hunting work to #200.
Renamed github/import -> github/repo. Made import a GET request.
Refactor the homepage view. Move search/sort/pagination to querystring
Zren commentedon Jun 24, 2014
I just imported /ZrenTest/TestUserScript/folder/folder/%E3%83%86%E3%82%B9%E3%83%88.user.js. On dev, it redirects okay.
On Production it redirects to: https://openuserjs.org/scripts/zrentest/%C6%B9%C8
%C6%B9%C8Instead of https://openuserjs.org/scripts/zrentest/%E3%83%86%E3%82%B9%E3%83%88
%E3%83%86%E3%82%B9%E3%83%88The script imports fine.
sizzlemctwizzle commentedon Jun 24, 2014
@Zren
That discussion has been moved to #200. And yes, I ran into the same problem myself (as has @Martii).
[-]UTF-8 MIME Type Declaration, BOMs, and other encoding standards[/-][+]UTF-8, MIME Type Declaration, BOMs, and other encoding standards[/+]Martii commentedon Jun 25, 2014
One thing I've noticed is there usually isn't a clear path of:
With a specific example... in our search routines Input is currently converted to a Working ISO 8859-1 "sort of" regular expression compatible syntax via deprecated
escapeand originally a plain ASCII regular expression match with some UTF-8 to match against the assumed UTF-8 data. e.g. Not much consistency here.Ideally Working encoding should always be UTF-8 but we know regular expressions have some issues (as do some other support routines expecting something else)... which is why they aren't always reliable to use with variant Input. Another reason why I opted out of using a re with sanitizing certain linked
@keys in another pr.For the docs we should clarify for prs that all transformations like this should be commented briefly in the source as to what the I/W/O is and perhaps a variable name identifier prefix ... unless it's in HTML/XML where the tags could of course denote this.
Remove the BOM from the only file I have found it on.
Uppercase according to [IETF](http://tools.ietf.org/html/rfc3629#sect…
Martii commentedon Jul 16, 2014
This is more or less been determined... closing.
Insert a BOM check for import and file upload
**Try** requesting UTF-16 and let GH do the transformation so *node* …