A collection of postprocessing helper functions and examples for Flat Data.
These examples and functions are written in Deno, a new language created by the same founders of Node.js and meant to improve on many aspects of Node.
Note: If you're noticing your scripts failing recently, try updating to
0.0.15. More info here
When writing a Flat Data Action, you can specify a path to a postprocessing Deno script that can manipulate the data downloaded by Flat even further.
- name: Fetch data uses: githubocto/flat@v2 with: http_url: http://api.coindesk.com/v2/bpi/currentprice.json # The endpoint to fetch downloaded_filename: btc-price.json # The http_url gets saved and renamed in our repository as btc-price.json postprocess: postprocess.ts # A postprocessing javascript or typescript file written in Deno
This is an example of a postprocessing script. Notice the use of Deno.args[0] to pass in the path of the downloaded_filename.
// The Flat Data postprocessing libraries can be found at https://deno.land/x/flat/mod.ts // Replace 'x' with latest library version import { readJSON, writeJSON } from 'https://deno.land/x/[email protected]/mod.ts' const filename = Deno.args[0] // equivalent to writing `const filename = 'btc-price.json'` const data = await readJSON(filename) // pluck a specific key off and write it out to a new file const newfile = `postprocessed_${filename}` await writeJSON(newfile, data.path.to.something)
Can be found in the examples folder. Once you install Deno you can run these examples with:
deno run -A examples/csv/csv-example.tsdeno run -A examples/csv/arquero-example.tsdeno run -A --unstable examples/image/image-example.tsdeno run -A examples/json/json-example.tsdeno run -A examples/sheets/sheets-example.tsdeno run -A examples/xlsx/xlsx-example.tsdeno run -A --unstable examples/zip/zip-example.ts
Deno can run javascript or typescript files, so you can easily convert any of these examples to javascript and run them in the same way:
deno run -A examples/csv/csv-example.js
While our examples use a Deno file to run postprocessing tasks, you can also use Python as specified in this example: https://github.com/pierrotsmnrd/flat_data_py_example. Thank you @pierrotsmnrd!
You can also use bash as specified in this example: https://github.com/aborruso/flat_data_bash_example. By @aborruso
The Flat Data postprocessing library can be found at: https://deno.land/x/flat/mod.ts
You can import and use these helper functions directly, or treat them as a starting point for writing your own postprocessing scripts.
readCSV(path: string, options?: ParseOptions): Promise<Record<string, unknown>[]>
Args:
- path: path to a local CSV file
- options: options for parsing the CSV file
Usage:
const csv = await readCSV('./path/to/file.csv')
writeCSV(path: string, data: Record<string, unknown>[] | string, options?: Deno.WriteFileOptions)
Args:
- path: path to a local CSV file
- data: string or object array to store
- options: options for writing the CSV file
Usage:
const data = [ { age: 70, name: 'Rick' }, { age: 14, name: 'Smith' } ] await writeCSV('./path/to/file.csv', data)
readTXT(path: string): string
Args:
- path: path to a local TXT file
Usage:
const text = await readTXT('./path/to/file.txt')
writeTXT(path: string, text: string, options?: Deno.WriteFileOptions): void
Args:
- path: path to a local TXT file
- text: text to write to file
- options: options for writing the TXT file
Usage:
await writeTXT('./path/to/file.txt', 'Content for the file')
readJSON(path: string): JSON
Args:
- path: path to a local JSON file
Usage:
const json = await readJSON('./path/to/file.json')
readJSONFromURL(url: string): JSON
Args:
- url: URL to a json file
Usage:
const json = await readJSON('www.url.com/file.json')
writeJSON(path: string, data: any, replacer?: any, space?: string | number): void
Args:
- path: path to a local JSON file
- data: data to store as JSON
- replacer: replacer function that transforms the results or an array of strings and numbers that acts as an approved list for selecting the object properties that will be stringified
- space: adds indentation, white space, and line break characters to the to the JSON text to make it easier to read.
Usage:
const data = { age: 40 } await writeJSON('./path/to/file.json', data) await writeJSON('./path/to/file-with-indentation', data, null, 2)
Our library relies on SheetJS, a library for parsing various spreadsheet formats. In addition to a simple readXLSX function you can access the core xlsx module by importing it directly.
import { xlsx, readXLSX } from 'https://deno.land/x/flat/mod.ts'
xlsx provides many more utility functions.
readXLSX(path: string): XLSX.WorkBook
Args:
- path: path to a local XLSX file
Usage:
const workbook = await readXLSX('./path/to/file.xlsx') const sheetData = workbook.Sheets[workbook.SheetNames[0]] const csvString = await xlsx.utils.sheet_to_csv(sheetData)
We recommend using a library like imagescript for more advanced image manipulation. See an example here.
readImageFromFile(path: string): Promise<Uint8Array>
Args:
- path: path to a local image file
Usage:
const bytes = await readImageFromFile('./path/to/image.jpeg')
readImageFromURL(url: string): Promise<{ bytes: Uint8Array; name: string; }>
Args:
- url: url string to an image
Usage:
const image = await readImageFromURL('www.url.com/image.jpg') const bytes = image.bytes const name = image.name
writeImage(imageBytes: Uint8Array, path: string): void
Args:
- imageBytes: a byte array
- path: path and name to write the image file
Usage:
await writeImage(bytes, './path/to/image.jpeg')
unZipFromFile( filePath: string, destinationPath: string | null = "./", options: any = {}, ): Promise<string | false>
Args:
- filePath: a path to a local zip file
- destinationPath: a folder path to unzip the files
- options: option.includeFileName can be true or false
Usage:
const result = await unZipFromFile('./path/to/folder.zip', './unzip/path') const output = result ? 'File unzipped successfully' : 'Error unzipping'
unZipFromURL( fileURL: string, destinationPath: string | null = "./", options: any = {}, ): Promise<string | false>
Args:
- filePath: a path to a local zip file
- destinationPath: a folder path to unzip the files
- options: option.includeFileName can be true or false
Usage:
const result = await unZipFromURL('www.url.com/file.zip', './unzip/path') const output = result ? 'File unzipped successfully' : 'Error unzipping'
removeFile(path: string): void
Args:
- path: path to a local file to delete
Usage:
await removeFile('/path/to/file.x')
Run all the tests:
deno test -A --unstable tests/*
Run separate tests
deno test -A --unstable tests/csv-test.ts