Invisible Characters on Stage!

Best practices in IT sometimes change, and the management of certain invisible characters in text files is one example. End of Line (EOL) characters have evolved, especially for version control systems integration purposes. In the same way, the Byte Order Mark (BOM) on Unicode text files is less and less used.

With 4D v19 R2, 4D has evolved smoothly to follow these best practices, giving you more flexibility along the way.


End of line (EOL)

You know that lines in text files are separated with End of Line character(s). On Windows, it’s a combination of two characters:

  • Carriage Return (#13, named CR) and,
  • Line Feed (#10, named LF).

On macOS, it was the legacy CR character.

Version control systems like Git don’t manage CR as EOL character, so there’s a need to use LF instead.

Byte Order Mark (BOM)

You also know that a Unicode text file can contain invisible heading bytes, named Byte Order Mark, that define the character set used.

4D followed the current best practices when introducing Unicode, by writing text files with a BOM by default. But as UTF-8 has nearly become the standard text file format, the BOM is less and less used.

NEW 4D convention ON TEXT FILES

From now on, following the trends, 4D writes text files without a BOM. And on macOS, 4D uses LF as EOL character. This is completely automatic for all files written by 4D, such as 4DSettings, 4dm, 4DForm, and so on.

When opening files in a previous 4D version, you won’t encounter any problem, even if they were written in 4D v19 R2, because these previous versions were able to open files without BOM and using LF as EOL character.

Compatibility settings

Want to follow 4D’s behavior? New compatibility settings are available that allow TEXT TO DOCUMENT and File.setText() to generate files without a BOM and use LF as EOL character on macOS when the optional “charSet” and “breakMode” parameters are missing. This is fully automatic; you don’t need to rewrite your source code to benefit from this new behavior.


The new compatibility settings apply to the commands TEXT TO DOCUMENT and File.setText() when they are used without the “charSet” or “breakMode” optional parameters.
No changes were made for the EOL character. It can still be forced using the “breakMode” parameter.

In the same way, you may want to define precisely if the files generated by TEXT TO DOCUMENT and File.setText() contain a BOM or not. To do this, define a Unicode character set ending with “-no-bom”, such as “UTF-8-no-bom” or “UTF-16-no-bom”,  to write a file without a BOM. To write a file with a BOM, you can still use the existing character sets without the suffix “-no-bom”, such as “UTF-8” or “UTF-16”.

• Product Owner •Damien Fuzeau has joined the 4D Product team in February 2019. As a Product Owner, he is in charge of writing user stories, then translating them to functional specifications. His job also entails making sure that the feature implementations delivered are meeting the customer needs.Damien is graduated from the University of Nantes in software engineering. He spent more than 23 years in its former company, first as developer (discovering 4D in 1997), and later as engineering manager and software architect. This company is a 4D OEM partner and deployed 4D based business softwares for thousands users, on hundreds servers. So, Damien is used to 4D development and deployment in a multi-language context.