From 7f47e48eabdb494b4026734e605e304ee3b4739b Mon Sep 17 00:00:00 2001 From: David Sherret Date: Tue, 12 Nov 2019 23:22:42 -0500 Subject: [PATCH 1/5] Add comment nodes rfc. --- rfcs/README.md | 2 +- ...C-0002 - Comment Nodes For All Comments.md | 195 ++++++++++++++++++ 2 files changed, 196 insertions(+), 1 deletion(-) create mode 100644 rfcs/RFC-0002 - Comment Nodes For All Comments.md diff --git a/rfcs/README.md b/rfcs/README.md index 366097428..ced285af4 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -10,4 +10,4 @@ This is a very loose RFC process. The implementation may slightly differ. ## In Progress -* None +* [RFC-0002](RFC-0002 - Comment Nodes For All Comments.md) - Comment nodes for all comments. diff --git a/rfcs/RFC-0002 - Comment Nodes For All Comments.md b/rfcs/RFC-0002 - Comment Nodes For All Comments.md new file mode 100644 index 000000000..ad1c0d6f9 --- /dev/null +++ b/rfcs/RFC-0002 - Comment Nodes For All Comments.md @@ -0,0 +1,195 @@ +# RFC-0002 - Comment Nodes For All Comments + +## Prerequisites + +* Understand comment ownership: https://github.com/Microsoft/TypeScript/wiki/Architectural-Overview#trivia +* Understand the difference between `getChildren()` and `forEachChild` in the compiler API. + +## Issues + +* [#737](https://github.com/dsherret/ts-morph/issues/737) - Parse out all comments as children. +* [#721](https://github.com/dsherret/ts-morph/issues/721) - `file.getDescendantsOfKind(SyntaxKind.SingleLineCommentTrivia)` does not return all comments + +## Problem + +It is difficult to deal with comments in ts-morph. RFC-0001 made it somewhat easier by introducing the concept of "comment nodes", which had "comment statement" and "comment member" implementations, but it didn't go far enough. There are still many comments that need to be parsed out each time from the compiler. + +## `Node#getChildrenWithComments()` + +This RFC proposes the following: + +1. Comment statements & members will be changed to "comment list statements" and "comment list members". +2. `Node#getChildren()` will return these comment lists instead of just a single comment node (which was the first comment on the line). +3. Add new `Node#getChildrenWithComments()` method that works the same as `Node#getChildren()`, but additionally parses out and returns comments as objects that implement the `ts.Node` interface. + +### Example + +Given the following code: + +```ts +// 1 +/*2*/ // 3 +/*4*/ test/*5*/; //6 +/*7*/ //8 +``` + +The current tree representation using `Node#getChildren()` in ts-morph is the following as per RFC-0001: + +``` +SourceFile + SyntaxList + CommentStatement (// 1) + CommentStatement (/*2*/) + ExpressionStatement + Identifier + SemiColonToken + CommentStatement (/*7*/) + EndOfFileToken +``` + +The new implementation of `Node#getChildren()` will return the following: + +``` +SourceFile + SyntaxList + CommentListStatement + SingleLineCommentTrivia + CommentListStatement + MultiLineCommentTrivia + SingleLineCommentTrivia + ExpressionStatement + Identifier + SemiColonToken + CommentListStatement + MultiLineCommentTrivia + SingleLineCommentTrivia + EndOfFileToken +``` + +Then `#getChildrenWithComments()` would return the following: + +``` +SourceFile + SyntaxList + CommentListStatement + SingleLineCommentTrivia + CommentListStatement + MultiLineCommentTrivia + SingleLineCommentTrivia + MultiLineCommentTrivia + ExpressionStatement + Identifier + MultiLineCommentTrivia + SemiColonToken + SingleLineCommentTrivia + CommentListStatement + MultiLineCommentTrivia + SingleLineCommentTrivia + EndOfFileToken +``` + +## Comment Node + +Comment nodes are any single line or multi-line comment in the file. They will work the same as the comment nodes described in RFC-0001. The only difference now is that any comment can be represented by a comment node and there is now an introduction of "comment lists". + +## Comment Lists + +Comment lists are a collection of comments located in the statements of a node or members of nodes like interfaces/classes/etc. that are on one line without any other tokens on that line. + +They show up in methods like `#getStatementsWithComments()`, `#getChildren()`, and the new `#getChildrenWithComments()`. For example, the following is a comment list: + +```ts +/*1*/ /*2*/ //3 +``` + +Tree representation: + +``` +CommentListStatement + MultiLineCommentTrivia (/*1*/) + MultiLineCommentTrivia (/*2*/) + SingleLineCommentTrivia (//3) +``` + +As described in RFC-0001 about comment statements & members, this is done to allow inserting before and after certain comments. + +### JS Docs + +JS docs are special and are not comment lists. They are included as children of the declaration they describe in the compiler API. + +Additionally, any comments that appear after a JS doc and before the declaration start are not comment lists. + +For example the following code: + +```ts +// 1 + +/** My function. */ +//2 +function myFunction() {} +``` + +Has the following tree when using `#getChildrenWithComments()`: + +``` +SourceFile + SyntaxList + CommentListStatement + SingleLineCommentTrivia + FunctionDeclaration + JSDocComment + SingleLineCommentTrivia + FunctionKeyword + Identifier + OpenParenToken + SyntaxList + CloseParenToken + Block + OpenBraceToken + SyntaxList + CloseBraceToken + EndOfFileToken +``` + +### Types + +```ts +export enum CommentListKind { + Statement, + ClassElement, + TypeElement, + ObjectLiteralElement, + EnumMember +} +``` + +Base type: + +```ts +interface CompilerCommentList implements ts.Node { + commentListKind: CommentListKind; + comments: CompilerCommentNode[]; +} +``` + +### `ts.SyntaxKind` choise + +The major downside here is that unlike comment nodes which use either `ts.SyntaxKind.MultiLineCommentTrivia` or `ts.SyntaxKind.SingleLineCommentTrivia`, there is no existing `ts.SyntaxKind` that works well for comment lists. The only available option seems to be to use a custom number or `ts.SyntaxKind.Unknown` (`0`), which I think is what I'm going to go with because that is in the set of `ts.SyntaxKind`. + +## Comments Inside Syntax Lists With Statements or Members + +For example: + +```ts +{ + //1 + a; + //2 +} +``` + +The syntax list of the block ends at the semi-colon of `a;`, but when calling `#getChildrenWithComments()` on the syntax list, it should return all the comments up to the close brace token. + +## Comment children ownership + +Child comments are any comment that is found within `Node#.getStart(true)` (`true` meaning the start pos including js docs) and `Node#end` where no descendant node is true for that condition. The only exception to this is syntax lists with statements or members as described above. \ No newline at end of file From a655e4a6ab2d75b808ce5d98f1d1876a68bf5fa5 Mon Sep 17 00:00:00 2001 From: David Sherret Date: Thu, 14 Nov 2019 22:03:54 -0500 Subject: [PATCH 2/5] More examples. --- rfcs/README.md | 5 --- ...C-0002 - Comment Nodes For All Comments.md | 36 ++++++++++++++++--- 2 files changed, 31 insertions(+), 10 deletions(-) diff --git a/rfcs/README.md b/rfcs/README.md index ced285af4..aa4e0c22a 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -4,10 +4,5 @@ This is a very loose RFC process. The implementation may slightly differ. [Issues](https://github.com/dsherret/ts-morph/issues?utf8=%E2%9C%93&q=label%3A%22requires+proposal%22+) -## Implemented - * [RFC-0001](RFC-0001 - Inserting Into Statements Handling Comments.md) - Inserting into statements handling comments. - -## In Progress - * [RFC-0002](RFC-0002 - Comment Nodes For All Comments.md) - Comment nodes for all comments. diff --git a/rfcs/RFC-0002 - Comment Nodes For All Comments.md b/rfcs/RFC-0002 - Comment Nodes For All Comments.md index ad1c0d6f9..4dc8c96eb 100644 --- a/rfcs/RFC-0002 - Comment Nodes For All Comments.md +++ b/rfcs/RFC-0002 - Comment Nodes For All Comments.md @@ -12,14 +12,16 @@ ## Problem -It is difficult to deal with comments in ts-morph. RFC-0001 made it somewhat easier by introducing the concept of "comment nodes", which had "comment statement" and "comment member" implementations, but it didn't go far enough. There are still many comments that need to be parsed out each time from the compiler. +For programmatic refactoring, it is very important to know about comments. In the past, all comments had to be requested via `Node#getLeadingCommentRanges()` and `Node#getTrailingCommentRanges()`. These returned objects would be forgotten after each manipulation and it was difficult to tell how these comments fit together with the nodes around them. + +RFC-0001 made it somewhat easier by introducing the concept of "comment nodes", which had "comment statement" and "comment member" implementations, but it didn't go far enough. There are still many comments that need to be requested on demand from `#getLeading/TrailingCommentRanges()`. ## `Node#getChildrenWithComments()` This RFC proposes the following: 1. Comment statements & members will be changed to "comment list statements" and "comment list members". -2. `Node#getChildren()` will return these comment lists instead of just a single comment node (which was the first comment on the line). +2. `Node#getChildren()` will return these comment lists instead of just a single comment statement/member. 3. Add new `Node#getChildrenWithComments()` method that works the same as `Node#getChildren()`, but additionally parses out and returns comments as objects that implement the `ts.Node` interface. ### Example @@ -111,6 +113,19 @@ CommentListStatement SingleLineCommentTrivia (//3) ``` +...and the following lines are all not comment lists: + +```ts +/*1*/ a; // these two lines have tokens, so not comment lists +test/*3*/; /*4*/ // 5 +class test +// not comment lists because these comments do not appear +// in the position of a statement or member +{ + +} +``` + As described in RFC-0001 about comment statements & members, this is done to allow inserting before and after certain comments. ### JS Docs @@ -167,16 +182,27 @@ Base type: ```ts interface CompilerCommentList implements ts.Node { + kind: ts.SyntaxKind.Unknown; commentListKind: CommentListKind; comments: CompilerCommentNode[]; + // ...etc.. +} +``` + +Example comment list type: + +```ts +export declare class CompilerCommentStatement extends CompilerCommentList implements ts.Statement { + _statementBrand: any; // this brand is from ts.Statement + commentListKind: CommentListKind; } ``` -### `ts.SyntaxKind` choise +### `kind` property value The major downside here is that unlike comment nodes which use either `ts.SyntaxKind.MultiLineCommentTrivia` or `ts.SyntaxKind.SingleLineCommentTrivia`, there is no existing `ts.SyntaxKind` that works well for comment lists. The only available option seems to be to use a custom number or `ts.SyntaxKind.Unknown` (`0`), which I think is what I'm going to go with because that is in the set of `ts.SyntaxKind`. -## Comments Inside Syntax Lists With Statements or Members +## Comments inside syntax lists with statements or members For example: @@ -192,4 +218,4 @@ The syntax list of the block ends at the semi-colon of `a;`, but when calling `# ## Comment children ownership -Child comments are any comment that is found within `Node#.getStart(true)` (`true` meaning the start pos including js docs) and `Node#end` where no descendant node is true for that condition. The only exception to this is syntax lists with statements or members as described above. \ No newline at end of file +Child comments are any comment that is found within `Node#.getStart(true)` (`true` meaning the start pos including js docs) and `Node#end` where no descendant node is true for that condition. The only exception to this is syntax lists with statements or members as described above. From 66b896cd1d1cf9bdd2a738291f26108eb9466366 Mon Sep 17 00:00:00 2001 From: David Sherret Date: Thu, 14 Nov 2019 22:04:40 -0500 Subject: [PATCH 3/5] Fix type name. --- rfcs/RFC-0002 - Comment Nodes For All Comments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/RFC-0002 - Comment Nodes For All Comments.md b/rfcs/RFC-0002 - Comment Nodes For All Comments.md index 4dc8c96eb..7877e23aa 100644 --- a/rfcs/RFC-0002 - Comment Nodes For All Comments.md +++ b/rfcs/RFC-0002 - Comment Nodes For All Comments.md @@ -192,7 +192,7 @@ interface CompilerCommentList implements ts.Node { Example comment list type: ```ts -export declare class CompilerCommentStatement extends CompilerCommentList implements ts.Statement { +export declare class CompilerCommentListStatement extends CompilerCommentList implements ts.Statement { _statementBrand: any; // this brand is from ts.Statement commentListKind: CommentListKind; } From 4b599c174e39bc34ba00bfea39b724e79795a94c Mon Sep 17 00:00:00 2001 From: David Sherret Date: Thu, 14 Nov 2019 22:24:02 -0500 Subject: [PATCH 4/5] More examples. --- ...C-0002 - Comment Nodes For All Comments.md | 21 +++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/rfcs/RFC-0002 - Comment Nodes For All Comments.md b/rfcs/RFC-0002 - Comment Nodes For All Comments.md index 7877e23aa..c52cad067 100644 --- a/rfcs/RFC-0002 - Comment Nodes For All Comments.md +++ b/rfcs/RFC-0002 - Comment Nodes For All Comments.md @@ -96,7 +96,7 @@ Comment nodes are any single line or multi-line comment in the file. They will w ## Comment Lists -Comment lists are a collection of comments located in the statements of a node or members of nodes like interfaces/classes/etc. that are on one line without any other tokens on that line. +Comment lists are a collection of comments located in the statements of a node or members of interfaces, class declarations & expressions, type elements, object literals, and enums. They must be on one line without any other tokens on that line. The exception is when a close brace token directly follows them and no token preceeds them. Additionally, comments that are between a jsdoc and the declaration it describes are children of the class declaration and therefore not comment lists. They show up in methods like `#getStatementsWithComments()`, `#getChildren()`, and the new `#getChildrenWithComments()`. For example, the following is a comment list: @@ -113,7 +113,21 @@ CommentListStatement SingleLineCommentTrivia (//3) ``` -...and the following lines are all not comment lists: +**Examples of comment lists** + +There is one comment list per line with a comment: + +```ts +// 1 +/* 2 */ // 3 +class Test { + // 4 + prop; + // 5 +/* 6 */ } +``` + +**Examples of comments that aren't comment lists** ```ts /*1*/ a; // these two lines have tokens, so not comment lists @@ -124,6 +138,9 @@ class test { } + +/* not a comment list because it ends +on the same line as a token */ b; ``` As described in RFC-0001 about comment statements & members, this is done to allow inserting before and after certain comments. From 31997900ce4d06da74551ecee462df364da67939 Mon Sep 17 00:00:00 2001 From: David Sherret Date: Fri, 15 Nov 2019 00:09:37 -0500 Subject: [PATCH 5/5] Include information about end of file JS doc. --- ...C-0002 - Comment Nodes For All Comments.md | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/rfcs/RFC-0002 - Comment Nodes For All Comments.md b/rfcs/RFC-0002 - Comment Nodes For All Comments.md index c52cad067..24ac5c7f2 100644 --- a/rfcs/RFC-0002 - Comment Nodes For All Comments.md +++ b/rfcs/RFC-0002 - Comment Nodes For All Comments.md @@ -233,6 +233,37 @@ For example: The syntax list of the block ends at the semi-colon of `a;`, but when calling `#getChildrenWithComments()` on the syntax list, it should return all the comments up to the close brace token. +## JSDocs at end of file + +In the compiler api, the following code: + +```ts +/** js doc */ +// test +``` + +Returns the following when calling `#getChildren()` on the nodes: + +```ts +SourceFile + SyntaxList + EndOfFileToken + JSDocComment +``` + +In this case though, it is still useful to insert around jsdocs at the end of the file so `#getChildrenWithComments()` will return the following: + +```ts +SourceFile + SyntaxList + CommentListStatement + JSDocComment - Same reference + CommentListStatement + SingleLineCommentTrivia + EndOfFileToken + JSDocComment - Same reference +``` + ## Comment children ownership Child comments are any comment that is found within `Node#.getStart(true)` (`true` meaning the start pos including js docs) and `Node#end` where no descendant node is true for that condition. The only exception to this is syntax lists with statements or members as described above.