Tuesday 6 June 2017

NodeJS: Need idiomatic: Read files in dir, concatenate, transform, write

I'm trying to write a content manglement program in Node. I'm an old Ruby / Perl / Shell hand for many years, and I can't seem to get simple code that works in those languages to look similarly, eh, simple, in Node.

Task: Find all the *.md files, read them (in ls order), transform them, and bracket them with a header comment and a footer comment. The files, in order, have a pieces of Markdown that, when assembled and transformed, are a sensible HTML doc. Here's a shell implementation:

echo '<!-- Generate at:' $(date) ' -->' $(ls *.md |xargs cat|markdown)'<!-- Copyright Mumble-demo Inc. -->'

Produces the desired HTML:

<!-- Generate at: Tue Jun 6 08:25:59 EDT 2017  --> <h1>This is a Markdown File</h1> <h2>Heading 1</h2> <p>Inside of markdown we can create many interesting items</p> <ul> <li>such</li> <li>as</li> <li>lists</li> </ul><!-- Copyright Mumble-demo Inc. -->

Ruby is similarly reasonable...

#!/usr/bin/env ruby
require 'kramdown'

HEADER = "<!-- Generated at #{Time.now} -->\n"
FOOTER = "\n<!-- Copyright Mumble-demo Inc. -->"

OUTPUT = File.open("./output", "w")

results =  Dir.glob("*.md").map { |f| File.open(f).readlines.join() }.reduce(:+)
OUTPUT.print(HEADER, Kramdown::Document.new(results).to_html, FOOTER)

But I can't figure out how to do this in Node in a Way That Feels Right(™)

A Way That Feels Wrong(™) is with the synchronous interfaces:

const fs = require("fs")
const marked = require("marked")

const HEADER = `<!-- Generated at ${new Date()} -->\n`
const FOOTER = `\n<!-- Copyright Mumble-demo Inc. -->`

fs.readdir(".", (err, files) => {
  if (err) throw err;
  let markdownFiles = files.filter((f) => f.endsWith(".md"))

  let content = markdownFiles.reduce((memo, fileName) => {
    return memo + fs.readFileSync(fileName, 'utf8')
  }, "")

  let contentString = [HEADER, marked(content), FOOTER].reduce((m, i) => m + i, "")
  fs.writeFileSync("derp", contentString);
  console.log(contentString);

})

A Way That Feels Right But That I Can't Get To Work(™) is:

  1. Build read streams
  2. Pipe them to markdown transform streams
  3. Open an output stream and redirect transformed data to it

Great news is – this approach works until it comes time to put the header comments in at the top and the bottom. They live in the code, not on the filesystem so I can't "just add" them as another file to stream, sans transform, into the output stream. Most approaches wind up producing: header, footer, streamed data

Obviously the pipe()-work works asynchronously and the footer print fires before the read + transform work is done. I've tried horrible (and broken) Promise chains that ultimately did not work.

One alternate approach would be to turn the header and footer into streams (seems weird...) and flow them into the output stream as well (seems really weird).

I've stumped several seasoned developers with this...surely we're missing some common idiom here or is it actually this hard to do this simple task simply in Node?



via Steven Harms

No comments:

Post a Comment