Shape Suffixes – Good Coding Style

Character.AI has implemented a coding style convention at their organization since 2022, which involves using single-letter names to represent logical dimensions of tensors, such as 'B' for batch size and 'L' for sequence length. This approach is demonstrated in an example Transformer code using PyTorch, where tensor names like 'input_token_id_BL' and 'hidden_BLD' are used. By appending these dimension-suffixes to tensor names, the code becomes more informative and easier to understand.

The use of shape suffixes is a response to the complexity of modern deep learning models, which often involve high-dimensional tensors with multiple dimensions. This convention can be applied to various deep learning frameworks, including PyTorch and JAX. Character.AI's adoption of this convention reflects the importance of code readability and maintainability in the development of complex AI systems. As AI models continue to grow in complexity, such coding style conventions can help improve collaboration and reduce errors among development teams.

The implications of this convention are particularly relevant for organizations working on large-scale AI projects, where code readability and maintainability are crucial. By adopting this convention, developers can write more readable and self-explanatory code, reducing the need for extensive documentation and comments. However, the effectiveness of this approach depends on its adoption and consistency across the organization, and it may require additional effort to document and maintain the dimension-suffixes used in the codebase.