Repository navigation
Fix UTF-8 Support #200
Description
@OpenUserJs/admin @OpenUserJs/backend @OpenUserJs/frontend
Related to #198 and a few other bugs, I just wanted to consolidate identifying and fixing a type of bug that might be common inside of OUJS. UTF-8 characters sometimes break things in production. It doesn't happen in development for some reason. It mostly has to do with encoding going wrong (or the wrong encoding function being used) during a redirect, but it might happen it other areas that have yet to be uncovered.
So far the errors I've been able to verify in production are:
- Import using GitHub (UTF-8 in GitHub path)
- Edit script description (UTF-8 in scriptname)
@Martii
Were there any others? Any other script manipulation I tried in production seems to work.
This is a blocking bug. I will not accept any unrelated PRs, except bug fixes, until this issue is resolved. Update: All issues that been discovered have been fixed, but there may be lingering undiscovered bugs so I'm leaving this issue open until we're confident this has been resolved.
Your mission (should you choose to accept it): Use UTF-8 characters in production in any way you can possibly imagine, and report back here with your results in the comments. This way we don't waste effort testing something twice and can identify the problem areas that need to be fixed.
Activity
Refactor the homepage view. Move search/sort/pagination to querystring
Renamed github/import -> github/repo. Made import a GET request.
Martii commentedon Jun 24, 2014
Not that I have seen yet. I could delete my affected scripts but not show the source or edit the source on production.
Most notably on the POST routines... e.g. they do change the data store on production but come back with ISO 8859-1 string redirects on dev (which means they work as expected but maybe not correctly) and UTF-8 string redirects on production... which is very peculiar.
See also:
sizzlemctwizzle commentedon Jun 24, 2014
Could you give me a url? I haven't been able to reproduce.
Wait a minute... Since they're redirections and not views, we must not be setting the character set correctly.
Martii commentedon Jun 24, 2014
I've subsequently deleted my affected scripts... on resync the issue of being non-editable isn't present but the 400 is... I didn't want orphaned scripts on OUJS currently if a fix shows up... I can infer that we were/are storing in ISO 8859-1 instead of UTF-8 in the DB I think. If you pick my Hello World script (only one in
UserScriptsrepo) it should do it as of testing earlier this morning. I haven't retried it with your recent commits and possible deployment yet.EDIT: You might try in jesus2099's account with his scripts (specifically this one that had source before DB Migration) that have Unicode in them too. I can't view Source Code on a few of them which is how I detected which ones of mine to delete.
EDIT: CONFIRMED pro importing my Hello World script causes the 400 with new tab on current, as of just a moment ago, still.
Martii commentedon Jun 24, 2014
btw production appears to be goofed up right now... with CSS at least. :\ This is challenging to find where things should be vs where they are. ;)
Zren commentedon Jun 24, 2014
Getting
I left out
type="text/css"in the<link>tags, but it was working without them so I've got no idea why it just broke.sizzlemctwizzle commentedon Jun 25, 2014
I was trying something out in production. Reverted. Should be fixed now.
Martii commentedon Jun 25, 2014
Later those should probably go back in from where they left. MIME types are important to maintain with client side.
It is. :) Thanks.
sizzlemctwizzle commentedon Jun 25, 2014
I've identified 7 scripts that are currently in production but can't be installed because of a fuckup in the AWS code I used to get rid of url namespaces.
https://openuserjs.org/scripts/jesus2099/kasi._PLAIN_TEXT_LYRICS_歌詞コピー_純文本歌詞https://openuserjs.org/scripts/©TriMoon™/FB-App_Widescreenhttps://openuserjs.org/scripts/DitherSky/Skype-линкификаторhttps://openuserjs.org/scripts/AMZMA/Yet_Another_Download_Helper_for_Various_FileHosting,_Just_Helper_Dengan_Taburan_Kacang_Almond_Diselimuti_Coklat_Belgia_Yang_Langsung_Meleleh_Bila_Masuk_Di_Mulut._[Argggghhh}_(╯°Д°）╯︵_ɯuᴎƃɐɯhttps://openuserjs.org/scripts/DutyCorp/Duty™_Enginehttps://openuserjs.org/scripts/jesus2099/JASRACへの直リンクhttps://openuserjs.org/scripts/Kelseydehaan/iBOOD_Autosubmit_to_sofortÜberweisung_(DE)(no_longer_PayPal)This issue will not occur going forward. It is isolated to these scripts. I should be able to fix this soon.
Martii commentedon Jun 25, 2014
There's also may be some on dev that I saw last night in my account too. Those are totally orphaned from what I can tell... e.g. can't delete them because no homepage access.
I haven't check today with current HEAD yet.from my dev script listings page. These were orphaned during some of the past commit fixes including dev DB migration from your discussion.
sizzlemctwizzle commentedon Jun 25, 2014
You're pretty much on your own in dev. I might just wipe out all existing script data.
23 remaining items
cvzi commentedon Sep 9, 2014
I found an error with a UTF-8 file (and I hope this is the right place to report it)
I tried to update one of my script via Upload Script on https://openuserjs.org/user/add/scripts
After the file was upload I got a 302 redirect back to https://openuserjs.org/user/add/scripts instead of the source code being updated.
I think that the problem was the BOM which was added automatically by Windows' notepad.exe when I edited the file.
Martii commentedon Sep 10, 2014
Tested BOM at https://github.com/Martii/UserScripts/blob/master/src/RFC%202606%C2%A73/BOM%20test/RFC%202606%C2%A73%20-%20BOM%20test.user.js
Notepad isn't known for adding BOMs but it's also gone through a lot of changes. Notepad++ on the other hand is how this one was crafted by changing to Encode with UTF-8 (which includes the BOM and verified in a hex editor starting file with
EF BB BF)Any chance that you can upload that to a GitHub (GH) repository so I can test that file instead of one that I created? Don't do a gist as I'm not sure that will maintain the BOM at all.
And yes this is the correct place. :)
cvzi commentedon Sep 10, 2014
I tried to upload your testfile to oujs.org and it worked. Maybe my files are corrupt in another way.
I overwrote the original file, but I was able to reproduce the error. I've done this with Windows 7.
Create a new file with Notepad++ and set Encoding to UTF8 without BOM.
Add at least one 2-Byte character (I used a German umlaut ö) and save the file.
Now open the file with notepad.exe and just save it.
And now the file has the
EF BB BFBytes. And I cannot upload it to OJUS.Here is my test file:
https://github.com/cvzi/Userscripts/tree/master/UTF8BOM_test
When I try to upload
UTF8 BOM test.user.json OUJS I get the behavior as mentioned before.When I try to import it via OUJS GitHub import I get this error page:
400 | Specified file does not contain a userscript header.Insert a BOM check for import and file upload
Martii commentedon Sep 10, 2014
@cvzi
I've confirmed it with your file, including hex editor check, and have a PR that will probably get merged tomorrow. Needs review time for a little while then we'll get you taken care of. Thank you very much for your report. :)
Some misc client side jQuery fixes
Fix for no `@name` scripts
**Try** requesting UTF-16 and let GH do the transformation so *node* …
Fix flagging and AuthAs
Ensure UTF-8 in the output stream for script source
Cleanup admin controller file
More encoding and controller symmetry
Repair issue count
Fix some encoding issues
Fix some encoding issues (#1855)