Tuesday, November 10, 2015

Monadic templating in C# - Part 2

In the previous article (Part 1), I had introduced a prototype of a monadic framework for generating formatted output (e.g. source code) from some complex, hierarchical set of data (e.g. Abstract Syntax Tree of some input language). This second part picks up where the first part left, the explanation of the actual templating by utilizing the string interpolation feature of C# 6.0.


Generating formatted output with lightweight templates has never been easier than with the string interpolation feature of C# 6.0. This feature renders some (used to be very useful) templating libraries, e.g. SmartFormat obsolete. String interpolation basically means that one can insert arbitrary expressions into a string by decorating it with curly brackets. We can finally forget String.Format and the numbered placeholders...

In this article, first, string interpolation is introduced briefly by contrasting it with some earlier approaches. Following that I'll show how to marry string interpolation with the previously introduced monadic generators. 

Lightweight string templating

In the past we had two options for lightweight templates. The most obvious choice was the ubiquitous String.Format. It gets a template in its first argument, and uses varargs for passing the objects referenced by the template (the exact same idea used in C since the seventies). The template language is simplistic, references are indexes of the arguments wrapped into curly brackets (if one wants to output curly bracket characters, those must be duplicated for escaping): 

String.Format("function {0}({1}){{{2}}}", name, arguments, body)

It is a very unpractical approach as it is very hard to read. It is mainly because there of the indirection in the template and that the indirection is achieved by numbered indexes. Numbered indexes are great for computers, but highly demotivating for humans.

A better approach is what used by e.g. SmartFormat, named placeholders:

Smart.Format("function {name}({arguments}){{{body}}}", function)

Unfortunately, this feature is limited. It works only if the template has exactly one object argument (the names in the template correspond to the members of the object then). It helps a bit that anonymous types can be involved:

Smart.Format("function {name}({arguments}){{{body}}}", new {name, arguments, body})

It is not bad any more, only those indirect references wouldn't be there...

This is when string interpolation comes into the picture:

String res = $"function {name}({arguments}){{{body}}}");

It is just great. It directly provides strings, and arbitrary expression can be embedded in the placeholders. It is exactly what we need.

Customizing string interpolation

Previously I was not completely honest, string interpolation actually does not provide string directly, instead of an object of type FormattableString. This can be stringified through its ToString method by a custom IFormatProvider.

When we want to execute our generators (a kind of state monad), we need to provide an initial state.
Thus all we have to do to use Generator<String> types expressions with string interpolation, is to use a stateful IFormatProvider:

class GFormatProvider : IFormatProvider, ICustomFormatter
    private Context ctx;

    public GFormatProvider(Context ctx)
        this.ctx = ctx;

    public object GetFormat(System.Type formatType)
        if (formatType == typeof(ICustomFormatter)) return this;
        return null;

    public string Format(string format, object arg, IFormatProvider formatProvider)
        if (arg == null)
            return string.Empty;

        // This is why we need the covariant type variable
        if (arg is IGenerator<object>)
            arg = ((IGenerator<object>)arg).Run(ctx).Item1;

        if(arg is IFormattable)
            return ((IFormattable)arg).ToString(format, formatProvider);
            return arg.ToString();

The provider is instantiated with a Context (the modified Context is dropped on purpose, but it could be returned if required). The Format method is executed for every placeholder, and it speaks for itself. It's worthwhile to note though that this is where we exploit that IGenerator<T> is covariant. Otherwise, an arbitrary IGenerator<T> couldn't be cast to IGenerator<object>.

The only missing piece now is the utility method in Generator<T> to hide the custom  IFormatProvider:

public class Generator<T> : IGenerator<T>


    public static Generator<String> Template(FormattableString formattable)
        return Wrap(delegate (Context ctx)
            var provider = new GFormatProvider(ctx);
            var res = formattable.ToString(provider);
            return CovariantTuple<String, Context>.Create(res, ctx);


With this straightforward extension to string interpolation, our prototype monadic generator/templating library is ready to make experiences. I hope its complexity does not hide the beauty in this monadic approach.

And finally, as usual, the completely source cod can be downloaded from the following GitHub repository: https://github.com/tallcomponents/MonadicTemplate.

Monday, November 9, 2015

Basics of PDF graphics and how to edit

A question that I often hear is: "how can I change the graphics of my PDF such as replacing text with some other text or replace a logo with another logo?" In general this is not a good idea. PDF is not designed for editing; it is designed as an end format much like ink on paper. Nevertheless there may be circumstances - such as when you don't have access to the source format - when editing a PDF is a requirement. This article explains the basics of PDF graphics and how graphics can be edited if you really have to.

PDF Graphics

A PDF document contains various types of information such as metadata (author, title, etc.), form fields, navigational data such as bookmarks, annotations such as comments, and last but not least graphics. Graphics can roughly be divided into three categories: curves, text and images. The graphics on a PDF page are described by a sequence of operators. Operators can be divided into 3 groups: 
  1. draw operators that draw curves, text and images
  2. graphics state operators that do things such as selecting a font, selecting a color or transforming the coordinate system (more about that later)
  3. marked-content operators that associate high-level information with graphics but do not affect the appearance. I ignore this group for the purpose of this article.
Each operator takes zero or more operands. Here is a simple example that draws a straight red line:

150 250 m      % set the current point to (150, 250)
150 350 l      % append a straight line to (150, 350)
1 0 0 RG       % set the stroke color to red
S              % stroke the line

The operator follows the operands that are used by the operator. On the first line, operator 'm' uses operands 150 and 250. 

Here is an example that involves text:

/F1 24 Tf        % set font to F1 and font size to 24
100 100 Td       % move text position to (100, 100)
(Hello World) Tj % draw the text 'Hello World'

On the first line, operator Tf takes operands /F1 and 24. Operand /F1 is the name of a font. Without going into the full details, it suffices to say that /F1 can be resolved to an actual font either inside or outside the PDF document. 

And finally, here is a one-liner that draws an image:

/I1 Do           % draw image

Similar to selection a font, /I1 is resolved to an actual image inside the PDF document. 

Graphics state

As said, operators can be divided into draw operators and graphics state operators. When the operators are processed from top to bottom a graphics state is maintained. The graphics state operators change the graphics state and the result of draw operators are affected by the graphics state. In the first sample we saw that the RG operator changed the stroke color to red and the S operator draws a line using the current stroke color.

Other graphics state operators set the line width, dash pattern, fill color, font size etc. Finally, there are two special operators that, respectively, save (q) and restore (Q) the graphics state. Simply put: the restore operator changes the graphics state back to the state at the previous save operator. They appear pair-wise and can be nested.

Coordinate System

A crucial part of the PDF imaging model is the coordinate system. The coordinate system determines where on the page a given coordinate such as (150, 250) is located and what the extend of a size is. PDF defines different coordinate systems. The most important two are user space and device space.

Device space

The device space is determined by the output device such as a printer or display on which a PDF page is ultimately rendered. Let's say that we want to render a PDF page to a Windows bitmap at 300 DPI, then from a Windows development perspective, the device space has its origin at the top-left corner, the x axis points to the right, the y axis points downwards, and the length of a unit (a pixel) is 1/300 inch.

User space

As opposed to the device space, the user space is device independent. For every page, it is initialized such that its origin lies at the bottom-left corner, the x axis points to the right, the y axis point upwards and the length of a unit is 1/72 inch or 1 point. The coordinates in the above PDF operator examples are in user space.

Mapping user space to device space

How these coordinates are mapped or transformed to coordinates in the device space is defined by the current transformation matrix or CTM. Let's see how this would look in code:
// width and height of a Letter page
float width = 612; // 612 points = 8.5 inches
float height = 792; // 792 points = 11 inches

// output device is a 600 dpi bitmap
float dpi = 600;
Bitmap bitmap = new Bitmap((int)(width * dpi / 72), (int)(height * dpi / 72));

// 4 page corners in user pace 
PointF[] points = new PointF[] { 
   new PointF(0, 0),           // bottom-left corner
   new PointF(0, height),      // top-left corner
   new PointF(width, height),  // top-right corner
   new PointF(width, 0)        // bottom-right corner
   string.Join("; ", points.Select(p => string.Format("({0}, {1})", p.X, p.Y))));

// calculate the coordinates of the corners in device space
Matrix ctm = new Matrix();
// flip vertical axis
ctm.Scale(1, -1);
ctm.Translate(0, -bitmap.Height);
// resolution
ctm.Scale(dpi / 72f, dpi / 72);
   string.Join("; ", points.Select(p => string.Format("({0}, {1})", (int)p.X, (int)p.Y))));

Changing the user space

The CTM is part of the graphics state and it can be changed using the cm operator. The cm take six operands that represent a transformation matrix. Changing the CTM will affect subsequent draw operators as you will see in the following example.

We have a page that measures 200 pt by 200 pt. The following image shows the empty page with the user space coordinate system laid on top of it:

We draw a red square measuring 50 by 50 and a smaller blue square measuring 25 by 25 inside the red square like this:

Next we transform the user space by translating it by (50, 75). Note that this is done before the figure is drawn.

Finally, the user space is rotated 30 degrees like this:

So instead of transforming the squares, we transform the user space and then draw the squares inside that user space. Depending on where you are coming from, this may feel counter-intuitive.


From a development point of view, a sequence of operators is not a convenient format. E.g. you can not easily navigate to an image on the page and retrieve its position. Its properties depend on the accumulation of all previous operators so you would have to process all of them first. The same is true for text and curves. 

Changing a graphic, such as moving a single image or rotating a piece of text would be even harder because you would have to insert operators in such a way that they would only affect the targeted graphic.

PDFKit.NET allows you to extract all graphics on a page as a collection of shape objects. Internally it will do all the hard work of interpreting the operators, creating shape objects from draw operators and assigning properties that reflect the current graphics state. After extracting the shapes, you can remove shapes, insert new shapes and change their respective properties. When done, you can write the shapes back to a PDF page. This will in turn generate the required sequence of operators and operands.

Example: Replace a logo

To demonstrate the use of shapes to edit graphics, we are going to replace a logo. See below the images of the original PDF and the PDF after replacing the logo:

Here is all the code:

static void Main(string[] args)
   using (FileStream fileIn = new FileStream(
      "indesign_shortcuts.pdf", FileMode.Open, FileAccess.Read))
      Document pdfIn = new Document(fileIn);
      Document pdfOut = new Document();

      foreach (Page page in pdfIn.Pages)
         ShapeCollection shapes = page.CreateShapes();

         // add modified shapes to the new document
         Page newPage = new Page(page.Width, page.Height);

      using (FileStream fileOut = new FileStream(
         "out.pdf", FileMode.Create, FileAccess.Write))

static void replaceLogo(ShapeCollection shapes)
   for (int i = 0; i < shapes.Count; i++)
      Shape shape = shapes[i];
      if (shape is ShapeCollection)
         // recurse
         replaceLogo(shape as ShapeCollection);
      else if (shape is ImageShape)
         ImageShape oldLogo = shape as ImageShape;

         ImageShape newLogo = new ImageShape("new-logo.png");
         newLogo.Transform = oldLogo.Transform;
         newLogo.Width = oldLogo.Width;
         newLogo.Height = oldLogo.Height;

         shapes.Insert(i, newLogo); 

Friday, November 6, 2015

Single Page ASP.NET Application for Splitting and Stitching PDF Documents

Download source code

This post shows a single page ASP.NET application that allows the user to:
  • Upload PDF documents
  • Drag and drop pages between PDF document and assemble new documents
  • Download the modifed PDF documents

The following technologies are used:

How the application works

Before going into the implementation details, let's take a look at the application itself:

Now that we have seen what the application does, let's take a detailed look at the code. The full project can be downloaded from here. In the text I will include code snippets and refer to the location of the source file in the project. Sometimes I will simplify the snippet for readability.

Design overview

The application is implemented as a single controller (/Controllers/HomeController.cs), a home view (/Views/Home/Index.cshtml) and a partial view (/Views/Home/_Panel.cshtml). Roughly, the application implements the following functionality:
  • Uploading a PDF and rendering the PDF as a list of pages
  • Dragging and dropping pages between documents
  • Downloading a new PDF document
The following diagram and pseudo-code shows the steps involved in uploading a PDF document and rendering the page thumbnails.

After uploading, each page is rendered as an img element that encodes its origin using data-guid and data-index attributes. When pages are dropped to another panel, this panel includes all information required to ask the server to create a new PDF document from the guid/index pairs. This is shown in the following diagram.

In the remainder of the article I will discuss all parts in detail by taking a look at the server and the client code.


For uploading, we use the jQuery File Upload Plugin. The HTML for the upload button can be found in /Views/Home/Index.cshtml and looks like this:
<span class="btn btn-success fileinput-button">
   <i class="glyphicon glyphicon-plus"></i>
   <span>Upload PDF...</span>
   <!-- The file input field used as target for the file upload widget -->
   <input id="fileupload" type="file" name="files[]" multiple>
Here is the client-side JavaScript event handler of the upload button:
   url: '@Url.Action("Upload", "Home")',
   dataType: 'html',
   sequentialUploads: true,
   done: function (e, data) {
The url argument points to the Upload action of the Home controller which can be found in /Controllers/HomeController.cs. This action method is quite simple. It just saves the uploaded PDF in an upload folder (using a new guid for the file name) and returns a partial view that displays the panel with the page thumbnails:
public ActionResult Upload()
   HttpPostedFileBase file = Request.Files[0];
   Guid guid = Guid.NewGuid();
   kit.Document pdf = new kit.Document(file.InputStream);
   file.SaveAs(Server.MapPath(string.Format("~/Upload/{0}.pdf", guid)));
   return PartialView("_Panel", new PanelModel() { 
      DocumentGuid = guid.ToString(), 
      Document = pdf 
The HTML of the returned view (discussed next) is available on the client side through the done callback of the fileupload function. This function passes it to the helper function addPanel that looks like this:
function addPanel(html) {
  var id = guid();
  // prepend html to div with id 'panels' (with a slide effect)
    .attr('id', id);
  // make the list of pages draggable
    connectWith: ".pageslist",
    stop: function (event, ui) {
It first prepends the HTML to the div with id 'panels' (with a slide effect). Next, it makes the list of pages draggable (discussed later).

Display the PDF document using a partial view

The Upload method returns partial view _Panel (see /Views/Home/Panel.cshtml) which is rendered from PanelModel. The Panel model is nothing more than a guid/document tuple. The document is an instance of the PDFKit.NET Document class. This third-party library used to programmatically combine pages from documents to create a new document. It is important to note that the Document instance only lives during the rendering of the partial view. It is discarded thereafter. The filename (the guid) is stored as an attribute in the HTML so that the PDF document can be loaded when needed. There is no in-memory server state whatsoever.
Here is the code of partial view _Panel:
@model PanelModel
<div class="panel">
  <div class="panelheader">
    <a href="#" class="closepanel pull-right"><i class=" fa fa-close"></i></a>
    <button type="submit" class="download btn btn-xs btn-primary">Download</button>
  <div class="pagesarea">
    <ul class="pageslist">
    @for (int i = 0; Model.Document != null && i < Model.Document.Pages.Count; i++)
      // enumerate the pages of the document
      Page page = Model.Document.Pages[i];
      // calculate the width of the thumbnail (18 dpi)
      int width = (int)((PanelModel.THUMBRES / 72f) * page.Width);
      int height = (int)((PanelModel.THUMBRES / 72f) * page.Height);
      // the src point to the Thumbnail action
      // attributes data-guid and data-index store the 
      // page origin for later retrieval
      <li class="ui-state-default">
        <img class="pagethumbnail" src="/Home/Thumbnail?d=@Model.DocumentGuid&i=@i" 
          width="@width" height="@height" data-guid="@Model.DocumentGuid" data-index="@i" />
The PDF panel uses the bootstap classes panel and panelheader. The panel header has a download button (discussed later) and a close button. I refer to the full source code for the close button. The download button is discussed later.
The body of the panel is an unordered list. The list items are the thumbnail images for the different pages. The following CSS styles the unordered list so there is no bullet and the items run from left to right:
ul.pageslist { 
    list-style-type: none; 
    float: left;
    margin: 0; 
    padding: 0; 
    width: 100%;
    min-height: 100px;
The object model of PDFKit.NET is used to enumerate the pages of the document. The interesting parts are the src, data-guid and data-index attributes of the img element. The src attribute points to a Thumbnail action method that dynamically renders the page thunbmnail. The URL includes the guid of the document and the page index. The Thumnail action is dicussed next.
The data-guid and data-index attributes of the img element make the page (thumbnail) self-describing. When the page is dragged (discussed later) to another panel, the data-guid and data-index are included and continue to identify the original document and page. This way, all state is maintained in the browser. How this state is used to download the new document is explained shorty.

Render page thumbnails

The src attribute of the page thumbnail image points to the following action method (see /Controllers/HomeController):
public ActionResult Thumbnail(string d, int i)
  // open the file in the upload folder identified by d
  using (FileStream file = new FileStream(
    Server.MapPath(string.Format("~/Upload/{0}.pdf", d)), 
    FileMode.Open, FileAccess.Read))
    // contruct a PDFRasterizer.NET document
    // and get the page at index i
    Document pdf = new Document(file);
    Page page = pdf.Pages[i];
    float resolution = PanelModel.THUMBRES;
    float scale = resolution / 72f;
    int bmpWidth = (int)(scale * page.Width);
    int bmpHeight = (int)(scale * page.Height);
    // render the page to a 18 DPI PNG bitmap
    using (Bitmap bitmap = new Bitmap(bmpWidth, bmpHeight))
    using (Graphics graphics = Graphics.FromImage(bitmap))
      graphics.ScaleTransform(scale, scale);
      // save the bitmap to the HTTP response
      bitmap.Save(Response.OutputStream, ImageFormat.Png);
  return null;
Looking at this code, arguments d and i are respectively, the file name and page index of the page to render. PDFRasterizer.NET is used to render this page to a gdi+ bitmap. this bitmap is then saved to the output stream as a png. the browser will display the thumbnail.

Drag and drop pages

For client side drag and drop, we use the sortable interaction of jQuery UI. The list of pages per panel is made sortable inside the addPanel function that we saw earlier, using the following code:
function addPanel(data) {
    connectWith: ".pageslist",
    stop: function (event, ui) {
When a page is dragged from one panel to another, no call is made to the server. All information required to download the new PDF document is stored client-side by means of the data-guid and data-index attributes of the thumbnail img elements.

Download new PDF document

Downloading the new document is the most interesting part. Before looking at the code, let me outline the 5 steps involved:
  1. CLIENT: The click handler of the download button enumerates the page thumbnails and creates an array of doc-guid/page-index tuples.
  2. CLIENT: The JSON representation of this array is POSTed to the Download action on the server.
  3. SERVER: Because a POST request cannot trigger the browser to open a file, it temporarily stores the JSON as a file and returns a guid identifying the JSON file.
  4. CLIENT: Next, the client makes a GET request to another Download action and passes the same guid.
  5. SERVER: The server reads the JSON from the temporary file, deletes the file, creates the PDF and writes it to HTTP response.
The click handler of the download button looks like this. Note that we use delegated events because download buttons are created dynamically. This handler includes all 3 client-side steps described above.
$(document).on('click', 'button.download', function () {
  // enumerate all the pages and create
  // an array of document guid/page index tuples
  pages = new Array();
  var id = $(this).parents('div.panel').attr('id');
  $('#' + id).find('img.pagethumbnail').each(function () {
    pages.push({ "Guid": $(this).attr('data-guid'), "Index": $(this).attr('data-index') });
  // POST array of tuples to download action
    type: "POST",
    url: '@Url.Action("Download", "Home")',
    data: JSON.stringify(pages),
    contentType: "application/json",
    dataType: "text",
    success: function (data) {
      // request the PDF
      var url = '@Url.Action("Download", "Home")' + '?id=' + data;
      window.location = url;
The server part consists of two action methods: One stores the JSON, the other creates and returns the PDF. This technique and alternatives are also discussed on StackOverflow. Here is the POST action:
public ActionResult Download(PanelPage[] pages)
  // create a JSON string from the PagePanel array
  string json = new JavaScriptSerializer().Serialize(pages);
  // save the JSON to ~/download/<newguid>.json
  string id = Guid.NewGuid().ToString();
  string path = Server.MapPath(string.Format("~/Download/{0}.json", id));
  System.IO.File.WriteAllText(path, json);
  // return the guid that identifies the JSON file
  return Content(id);
public class PanelPage
  public string Guid { get; set; }
  public int Index { get; set; }

Show/hide download button

The download button at the top of a panel is only shown if the panel has atleast one page, otherwise there is nothing to download. This is taken care of by the updatePanels function which is called whenever a new panel is added or when a page is dragged from one panel to another:
function updatePanels() {
  $('.panel').each(function () {
    if ($(this).find('.pagethumbnail').size() == 0) {
    else {

Tuesday, October 27, 2015

Monadic templating in C# - Part 1


When one is to generate formatted output from a complex hierarchical set of data, in our example, generating source code from an abstract syntax tree (AST), the task must be or practical to be divided into smaller subtasks, small generator functions which are responsible for producing output of a specific part of the data.

The output of the generator functions must be collected, then concatenated or combined together in a complex way using some templating technique. Speaking about string outputs, concatenation also must be done in an efficient manner using helper classes e.g. StringBuilder. In addition, the individual generator functions probably share some generation context with non-trivial scoping. They may generate warnings or errors along with the formatted output, which also must be collected. And the requirements can go on...

All of these requirements affect the calling convention between the individual generator functions, and it can be very complex even in a single case. The generation context must be passed, multiple return values must be received: generated output, warnings, modified context... It is also very hard to change the calling convention later as it easily pervades the whole source code. 

In this post we explain a framework for generating such formatted output using distributed generator functions. The framework is based on a technique heavily exploited in the functional programming world, called monads. With this technique arbitrary complex calling convention can be hidden in a small library, which can be extended later without affecting the actual generation logic.

In this first part of the post, the monadic generator functions are explained in details. The generator functions will be very lightweight in usage while can efficiently collect and concatenate the output of the generator functions substasks are delegated to. In our explanation example, an AST based source code generator, the generator functions share a generator context. The context is immutable and/thus the scope of the changes can be freely managed.

In the second part of the tutorial, the framework will be extended to provide lightweight monadic templating using the string interpolation feature of C# 6.0.

Throughout the tutorial, we stick with the immutable approach as we believe that it results in less error prone software in complex tasks. As Eric Lippert phrases in his blog post:
"ASIDE: Immutable data structures are the way of the future in C#. It is much easier to reason about a data structure if you know that it will never change. Since they cannot be modified, they are automatically threadsafe. Since they cannot be modified, you can maintain a stack of past “snapshots” of the structure, and suddenly undo-redo implementations become trivial. On the down side, they do tend to chew up memory, but hey, that’s what garbage collection was invented for, so don’t sweat it."
The source code of the example used throughout this post can be downloaded from
the following GitHub repository: https://github.com/tallcomponents/MonadicTemplate. The example is based on C# 6.0 features as string interpolation and static imports, thus one needs VisualStudio 2015 to be able to compile it.

Generator functions

In the following, let us suppose that we have the task already mentioned in the introduction, namely, generating source code given a hierarchical AST. We will use this example throughout the post.

First, let us see how a generator function would look developed in a naive way. We are basically doing string concatenations, so it must be done carefully to be efficient enough. Using StringBuilder is a trivial choice, the first approach could be to pass around a StringBuilder instance. Unfortunately this approach is not compositional, the result is messy, hard to read and maintain; templating cannot be easily achieved either, because of the lack of compositionality:

public void GenerateMethod(StringBuilder o, Method method)
   o.Append("function ");
   GenerateIdentifier(o, method.Name);
   GenerateArgumentList(o, method.Arguments);
   GenerateBlock(o, method.Body);

Having a string return type results in more readable code and it comes only with a small price of managing a local StringBuilder. Templating is also possible with this approach:

public string GenerateMethods(List<Method> methods)
   var o = new StringBuilder();

   foreach(var method in methods)
      var name = GenerateIdentifier(method.Name);
      var arguments = GenerateArgumentList(method.Arguments);
      var body = GenerateBlock(method.Body);
      o.AppendFormat("function {0}({1}){{{2}}}", name, arguments, body);
   return o.ToString();

It is getting more complicated though if generation context is introduced. To keep the solution compositional, the context is passed around in a "ref" parameter (another design could be to return a tuple of values from the generator functions as it would be done in a functional language):

private string GenerateMethods(ref Context ctx, List<Method> methods)
   var o = new StringBuilder();
   foreach(var method in methods)
       var name = GenerateIdentifier(ref ctx, method.Name);
       var arguments = GenerateArgumentList(ref ctx, method.Arguments);
       var body = GenerateBlock(ref ctx, method.Body);
       o.Append($"function {name}({arguments}){{{body}}}");
   return o.ToString();

Implementation details are getting pervades the source code now. If more features are added, e.g. generated warnings, writing generator functions are getting very tiresome and error prone. This is the point where functional programming "design patterns" come to the rescue.

Monadic generators

The technique we will use, called monad, was introduced by Phillip Wadler in his seminal paper Monads for functional programming in 1992.

According to Wikipedia:
"In functional programming, a monad is a structure that represents computations defined as sequences of steps: a type with a monad structure defines what it means to chain operations, or nest functions of that type together".
In other words, a monad is a data structure which helps to define steps of computations in a compositional way: basic computation steps can be composed together to an indistinguishable new step. A monad usually has two operations associated with it:
  1. return lifts a constant value into a monad;
  2. bind or >>= composes to computations into a new computation in a sequential manner. a >>= b means that a computation precedes b computation.
In our case the monad type will be the Generator<T> class. The is a bit more general than it was in the introductory example, but we want to enable the generator functions to be able to produce any value, not just strings. For practical reasons the T type variable must be covariant, thus an interface also must be utilized. It contains only one function to execute the monad (we will discuss this concept later):
interface IGenerator<out T>
   ICovariantTuple<T, Context> Run(Context ctx);
The actual Generator<T> class simply wraps a generator function, a delegate called GFun<T>. The wrapping can be done using the Wrap function. An arbitrary value can be lifted into the monadic context using the Return function. The bind operation is basically the + operator. This latter is a bit limited as it always return Generator<string>; this is enough for our purposes, the generic nature of the Generator<T> class will be exploited in the templating code.
    public delegate ICovariantTuple<T, Context> GFun<T>(Context ctx);

    public class Generator<T> : IGenerator<T>
        private readonly GFun<T> f;

        private Generator(GFun<T> f)
            this.f = f;

        private static Generator<T> Wrap<T>(GFun<T> f)
            return new Generator<T>(f);

        public ICovariantTuple<T, Context> Run(Context ctx)
            return f(ctx);

        public static Generator<T> Return(T val)
            return Generator<T>.Wrap(delegate (Context ctx)
                return CovariantTuple<T, Context>.Create(val, ctx);

        public static Generator<string> operator +(Generator<T> a, Generator<T> b)
            return Wrap(delegate (Context ctx)
                StringBuilder o = new StringBuilder();

                var Res1 = a.f(ctx);

                var Res2 = b.f(Res1.Item2);

                return CovariantTuple<String, Context>.Create(o.ToString(), Res2.Item2);


Basic examples

And this is how to use our monad (in the following we skip the class name from method invocations, e.g. Return("string") instead of Generator<string>.Return("string"); this can be done anyway using static imports). The simplest possible example is lifting a constant to the Generator monad:

Generator<string> res = Return("str");

Let us contemplate a bit on this. Return("str") has the type of Generator<string> which is just a wrapped delegate having an argument of type Context. That is, when a Generator class is created, only a closure is created, a kind of suspended computation which is not executed until the context argument is finally provided. This is where the Run method of the IGenerator<T> interface comes into the picture:

string str = res.Run(new Context()).Item1; // Item2 contains the modified context

Then again, when two generators are composed using the bind operation (+) the result is also a suspended computation, none of the parts are executed until the generation context is finally applied to the compound generator:

Generator<string> res = Return("str1") + Return("str2");

Running res results in a cascade of recursive executions of the embedded generators. The delegate related to the (+) operator takes the first generator, executes it using the provided context then executes the second computation using the context returned by the first computation (the values are concatenated using an internal StringBuilder).

string str = res.Run(new Context()).Item1; // == "str1str2"

Accessing to the context

For a more complex example we need to introduce two more methods for the Generator<T> class.
These methods enable to access to the current context:

public delegate Generator<T> GContextFun<T>(Context ctx);

public class Generator<T> : IGenerator<T>
   public static Generator<T> WithContext(GContextFun<T> gfun)
      return Generator<T>.Wrap(delegate (Context ctx)
         return gfun(ctx).Run(ctx);

   public static Generator<T> SetContext(Generator<T> gen, Context localctx)
      return Generator<T>.Wrap(delegate (Context ctx)
         var ret = gen.Run(localctx);
         return CovariantTuple<T, Context>.Create(ret.Item1, ctx);

The WithContext method wraps a GContextFun<T> into a generator; GContextFun<T> enable read-only access to the current generation context. The SetContext method enables to change the context with a local scope (for a given generator; the context returned by the generator is dropped in line 20). The following utility functions demonstrate their usage. It is supposed that the Context class contains the pair of GetProperty/SetProperty methods:
public Generator<string> GetContextProperty(string property)
   return WithContext(delegate (Context ctx)
       return ctx.GetProperty(property);

public Generator<T> SetContextProperty<T>(Generator<T> gen, string property, Generator<string> value)
   return WithContext(delegate (Context ctx)
       return SetContext(gen, ctx.SetProperty(property, value));

A more realistic example

Now we can have a quick glimpse into how real code, based on the Generator<T> class, would look like. The Template method used in the following example is still a debt of us. It is already uses a monadic extension of the string interpolation feature of C# 6.0, this will be introduced in the second part of this blog post series. Still, you can see how much eaiser to read it than the String.Format based version used in the introductory examples:

public Generator<string> GenerateBlock(Block block)
   return R("Method body for ") + GetContextProperty("method");

public Generator<string> GenerateMethod(Method method)
   var name = GenerateName(method);
   var arguments = GenerateArguments(method);
   var body = SetContextProperty(GenerateBody(method), "method", name);
   return Template($"function {name}({arguments}){{{body}}}");

Much nicer, isn't it?


As the previous example shows, using the introduced framework, we can concentrate on the pure "business" logic, the code is very readable and easy to extend. It is also efficient enough; F# also runs on CLR, it is prepared to deal with higher order functions efficiently. These are pros, but what are the cons?

First of all, it increases the conceptual complexity of the code a great deal. Implementation details are now hidden behind higher order functions; that can make the inexperienced hard to see what happens under the hood. Further disadvantage is that with this monadic approach we introduced a kind of lazyness. When a generator function is executed it just "prescribes" the computation steps, the actual execution is postponed until the monad is executed. This can make debugging a challenge in some cases.


We already used templating in the latest example. We continue with its explanation in the next part of this blog post series. 

Thursday, August 20, 2015

Adobe Reader requires Asian font packs for Western Documents

We recently encountered reports that Adobe Reader XI wants to install Asian language packs for files that got generated by our software and that are in fact, completely “western”.

Adobe Reader X did not require such a thing, so we wondered what had happened.
East Meets West

It turns out that this happens for documents that have non-embedded TrueType fonts. In PDF, one must incorporate some information for each font that is used in a document, even if it is not embedded. This can be done in various ways. In the past, we chose to incorporate so-called “CID”-style font information. This has the advantage that it can address the complete range of all Unicode characters, so in principle one can safely write out all Unicode text with such a font, without worrying about it any further (mixing Western, Asian, and even Klingon, provided that the fonts contains definitions for these glyphs).

One of the requirements for this type of font that the PDF specification imposes, is that it needs to have an encoding from a predefined sets of encodings. This limited set happens to just contain encodings that are classified as “Eastern” (Chinese, Japanese, Korean, etc.). There is however nothing particular “Eastern” about these. An encoding just specifies which sequence of bytes in a text maps to which glyph ID in the font. Languages are not really involved at this point.

And so we used these “Eastern” encodings for many years, and this has never lead to any issues having to needlessly install eastern language packs and the like. For all that really matters is not the encoding, but the characters that are actually used in the document.

Well, Adobe Reader XI changed all that. It now confuses our customers, prompting them to install an Asian language pack for documents that do not have a single eastern character in them. All because the encoding for the used CID fonts is “Eastern” (it has to be, because Adobe requires it for CID fonts).

To avoid this, it appears that we will have to start avoiding CID fonts for western documents. This means that we will have to investigate each piece of text, check whether the characters fit within a particular western single-byte code page, and output a TrueType font description with that single byte encoding. It also means that we may have to output multiple of these font descriptions into a document if not all characters fit within a single western single-byte code page.

It seems like an awkward step to take, now that Unicode should actually have solved all these code page troubles many years ago.

Tuesday, September 2, 2014

Why Adobe LiveCycle DRM is not supported?

Adobe LiveCycle Rights Management is a server-based security system that provides dynamic control over PDFs. It is a complex solution which, besides PDF, supports many document formats e.g. Microsoft Word.

Its key capabilities, according to their website, are

  • Restrict access of a document to certain individuals only.
  • Monitor the usage of a given document.
  • And most notably, revise usage rights or revoke access to documents after they have been distributed.

All of these dynamic capabilities require some kind of server based solution; the usage rights and the access policy is, instead of being part of the document, provided by some external policy server.

In practice, that requires the clients (e.g. a PDF viewer application) and the policy servers to implement a communication protocol to obtain decryption keys, access policy and so on.
Obviously, Adobe has the technology for a relatively safe communication between the client and the policy servers. However, it is not public technology, third party software cannot implement it.

If you encounter the exception "Unsupported encryption found" in any of TallComponent's PDF components, it is very likely that you stumbled upon a PDF treated with this DRM. All you can do is to double check whether this is the case. To check it, open your document with Adobe Reader (it might take some time as it has to connect the policy server to check the access restrictions). Open Security Settings, then select Permission Details:

The Security Method field contains the applied encryption algorithm. If it says "Adobe LiveCycle Rights Management", you are unfortunately bounded to Adobe Reader.

Tuesday, June 3, 2014

Printing PDF documents in WPF with a preview

Printing PDF documents in WPF applications

It is easy to print PDF documents in a application using the standard 'PrintDialog' in .NET. How to do so is described in this knowledge base article. However, the dialog being used is the standard one and is rather limited in it's functionally. It does not give you a preview either on how the document will look like when its printed and some common print options are missing.

Printing with a preview

What you may want instead is a preview of how the document will be printed on paper that is larger than the PDF document, like:

Or you may want to see what part will be printed on smaller paper, like:

Or, what will the output look like when the document is rotated to landscape and vertically centered and with automatic scaling to the printable area on the paper:

Extra print options

The preview is also useful when you want to print only one or a few pages of the document, like this:

Anyway, the extra print options like duplex printing, like shown above, should always be available to reduce the usage of paper sheets.

The WPF code sample

The code sample 'ScaledPrintingWPF' in PDFrasterizer.NET shows an implementation of the features above. It looks like this:

It is implemented using the following MVVM architecture:

The XAML view
A custom print dialog as shown above is implemented in XAML and has a menu to open a PDF file. It uses bindings for all printers settings and the preview box. It also has the usual code-behind part which contains only a small amount of code.

The view controller
The bindings connects the dialog to the view controller, which has the user interface logic and interfaces to the print model. 

The print model
The print model is where is starts to get more interesting, here is a list of what is does:
  • it gets a list of available printers and sets the selected one.
  • it gets a list of available paper formats and set the selected one
  • it gets a list of available paper sources and set the selected one
  • it creates a print ticket and makes any item of available to the view controller
  • it holds the opened PDF document
  • it renders the currently selected page of the PDF document, transform it according to the settings that the user has made, combines it with the printer paper and the printable area and scales it so it will fit in the preview image.
  • It presents the print ticket and a couple of other settings to the document paginator at the moment when the document is printed.
  • It passes the print progress information as delegates back to the view controller so the user can be notified on the print progress.
The document paginator
This renders the PDF document one page at the time and actually spools it to the printer queue along with the print ticket. It also notifies the print model on the progress being made.

Even more print options

Not all the possibilities of the printer ticket are implemented in this code sample, i.e. the settings for the staple and the page-order. However this can easily be implemented by just taking the duplex setting and replicate it to any other setting you might need. The duplex is a good example for these kind of extensions.


Implementing a such a WPF print dialog may not be that simple as there are a couple of hurdles you run into. For example the used DocumentPaginator class, which allows you to print large documents without too much memory usage can raise the infamous "FixedPage cannot contain another FixedPage" exception. Second, getting the transformations needed to scale and position the PDF in the preview tends to have a bit of complexity.

This code sample is part of the PDFRasterizer evaluation download.