Description

jrwdunham

opened on Aug 19, 2015

Issue body actions

The CSV mapreduce is taken from https://gist.github.com/xalakox/3026004 and I don't trust that code. For one thing, it's not clear what CSV "dialect" it's assuming.

In one CSV export I tested (taken from the FAQ of the Spreadsheet app), there were between 16 and 19 fields per line when parsed with Python's csv module (using the "excel" dialect). This is wrong since all lines should have the same number of fields.

The FAQ says "warning, csv is a lossy format, don't use this for backing up your data" but the problem is not that CSV is "lossy" but that a) we aren't explicit about what CSV dialect we're returning and b) we're using a suspect mapreduce function to create it.

Activity

1. cesine commented on Aug 20, 2015

   

   cesine

   on Aug 20, 2015

   Member

   More actions

   So what you are really complaining about here, isn't coming from your linguist side, its coming from your python side. You like data in tables, not json, and you need us to think of the CSV export as an API, and add tests to ensure valid output so it's more useful for developers? since you found the invalid input it would be best if you fix the csv export its in this file: ~~~https://github.com/FieldDB/FieldDB/blob/master/couchapp_dev/lists/asCSV.js~~~

   • find a proper csv library in javascript
   • write some tests for diverse input (see the Lexicon-spec.js for an example of loading data and running a map reduce on it
   • document your changes in the FAQ in a new section called "How do can I use my data from Python?"

   I'd expect you to be the first person to agree that mapping a tree to a linear order is a lossy transformation, no matter if you make the csv import use the "excel" dialect, its still going to lose all the hierarchal relationships between your fields, all their metadata, all your audio/video metadata, comments metadata, and any other info that isn't in the string representation of the fields.

   we just wanted to scare lab managers into not using CSV as a backup, if you're using the CSV as in input to a python process thats another story. we never expected anyone would prefer the CSV transformation over the actual data as in input to a python script, but it definitly makes sense if you're doing something with the IGT.
2. cesine

   assigned

   jrwdunham

   on Aug 20, 2015
3. cesine

   changed the title [-]CSV export has issues[/-] [+]CSV export should use the "excel" dialect for consumption as an API[/+] on Aug 20, 2015
4. jrwdunham commented on Aug 20, 2015

   

   jrwdunham

   on Aug 20, 2015

   MemberAuthor

   More actions

   This really has nothing at all to do with Python. I just happened to have used that programming language to analyze the CSV exports. CSV is a tabular format. It assumes a fixed number of columns and that each row will have a value for each column. asCSV.js simply doesn't produce valid CSV, or it does but its using some strange undocumented convention/dialect.

   I agree that CSV is lossy and I understand that the CSV export was kind of grudgingly added on (and I don't even like CSV either). But I compared several CSV exports to their JSON counterparts and I could find no instances of data loss of the type you're alluding to: every attribute value in the JSON was a string, even when I looked for attributes whose values I know are not strings underlyingly. So if you compare the CSV to the JSON exports that Spreadsheet produces you won't find data loss, but you will find that the CSV file probably won't open correctly in your Excel/OO/Numbers spreadsheet program. Moreover, this confusion over "lossy" in the documentation came from a user, not from me.
5. cesine commented on Aug 20, 2015

   

   cesine

   on Aug 20, 2015

   Member

   More actions

   joel you should tell us if something comes from a user, its normal we assume its coming from you if you add the issue! It sounded to me like you had a grumpy day and you were venting.

   FAQ wording has indeed deteriorated into inaccurate and confusing uselessness. The FAQ in the spreadsheet no longer links to the original json export (the pure data), but rather the flat object (same as is used for the csv) so both the csv and json exports are lossy and shouldnt be used for backups.

   More importantly the good news is that some user, somewhere, actually read that paragraph! yippee someone cares about exporting :)
   Can you take the original comment from the user in their context of what they were trying to do, and

   • figure out a way to fix the confusion in the documentation (preferably re-write that entire seciton, or minimally either put back the original pure json link or tell the user that both formats are not for backing up.) https://github.com/FieldDB/FieldDB/blob/spreadsheet_v2453/angular_client/modules/spreadsheet/app/locales/en/messages.json#L696
   • fix the csv export to whatever you want it to be. you have complete control over this... this one is the one in the export world, and its not the one you said you didnt trust. do you trust this one more? https://github.com/FieldDB/LinguisticDataConverters/blob/master/lists/get_data_as_csv.js

   You should still add a section in the FAQ about working with Python, many many RAs and linguists use Python, so this section will catch their eye. Even if the user was't trying to use python, some others will..
6. cesine

   added this to the Online export Release 1 milestone on Aug 20, 2015
7. cesine commented on Aug 25, 2015

   

   cesine

   on Aug 25, 2015

   Member

   More actions

   @jrwdunham louisa found and fixed an important bug in the spreadsheet app last week #2057 , so we need to re-deploy it. do you want me to do the FAQ updates for you?
8. jrwdunham commented on Sep 4, 2015

   

   jrwdunham

   on Sep 4, 2015

   MemberAuthor

   More actions

   @cesine I would appreciate it if you could do the FAQ updates.
9. cesine commented on Sep 5, 2015

   

   cesine

   on Sep 5, 2015

   Member

   More actions

   No problem, I think I checked it off in the last deploy so its done and deployed on prod. Can you take a look if you want to spot typos and thinkos?

Issue actions