Saving project source code without tokens

Commands, constants, tables and fields are stored with their tokens in the project source code files (4dm files). This allows 4D to rename them automatically. But sometimes, you would like these source code files to be stored without tokens for a better readability with a version control system or an external code editor, or for better code sharing between projects. Let’s see how to make 4D store source code without these tokens.

What are tokens?

This system allows 4D to automatically recognize a command, a constant, a table or a field in the legacy language even if these elements have been renamed.
A command name is followed by “:C” and the command ID, a constant by “:K” and the constant ID, a table by”:” and the table ID, a field by “:” and the field ID.

When opening the source code in the method editor, 4D automatically renames the elements according to their tokens. These tokens are hidden by 4D in the method editor, but if you use an external source code editor, they are visible.
Reading source code may be more difficult in external source code editor or version control systems because of these tokens.

Sample of a method with tokens:

How to disable them?

To make the reading easier, 4D now offers you the possibility to store the source code files without tokens.
You have a new 4D preference at your disposal to make your new projects writing source code files without tokens.

By default, the option is enabled so that there’s no change: your source code is still saved with tokens. But if you disable the option, the source code files of your new projects will be saved without tokens.

Sample of the same method without tokens:

What about my existing project?

When you disable the tokenization 4D preference, the 4DProject file of the new projects contains a boolean attribute named “tokenizedText” set to *false*.
Technically, you can add this attribute to the 4DProject file of an existing project, but it won’t delete the tokens in the existing source code files until they are re-saved by a modification in the 4D method editor or using the METHOD SET CODE command. Note that this attribute is usable since 4D v19 LTS.

How does it work?

You may know that in a project, all your source code is saved in English in the 4DM files, even if you develop using the French language.
When the 4D method editor opens a 4DM file, tokens are taken in account first. And then, the commands, constants, tables and fields are recognized by their text (and translated if you develop with the French language). So, even if your source code is stored without token, it’s still editable as usual in the 4D method editor!

Just don’t forget that if you disable the tokenization, you will have to perform yourself the command/constant/table/field renaming in legacy language, as it’s the usage with other development platforms.

Framework project

If you develop a framework project to be shared with another projects, you may encounter issues regarding the tables and fields with legacy code. You can’t limit yourself to copying the files from the framework to the project if it contains tables that don’t have the same IDs. For example, the [Setting] table of your framework may be #1, but #3 in a target project. If the copied source code files from the framework contains tokens, 4D will replace the [Setting] table by the one having the #1 ID in the target project!

In order to make source code sharing easier, you can disable token saving in your framework project. Then, after you copy the framework files in the target project, it will recognise the [Setting] table by its name and not its (potentially wrong) ID!