1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
|
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<html>
<body>
<span style="float:right">Tor Norbye <tor.norbye@sun.com></span>
<h2><a href="apichanges.html">API changes</a></h2>
<h1>GSF</h1>
<p>
GSF is a language infrastructure for NetBeans.
The purpose of GSF is to help create first-class editing support
for a group languages (like Ruby, Groovy, JavaScript, Python, Scala,
PHP, etc). By "first-class" editing support I mean all the features
that we currently have for Java in NetBeans. Deep editing support
including intelligent code completion, quick fixes and hints,
and so on.
</p>
<p>
The abbreviation
"GSF" is an historical artifact and a new name should probably be
assigned.
</p>
<h2>Philosophy</h2>
<p>
The GSF API is intended for "language plugins". These are modules
which add support for new languages (such as Ruby, JavaScript,
HTML, CSS and so on) to NetBeans.
The <b>key philosophy</b> behind GSF is:
<blockquote style="background-color: #ffddff; color: black; padding: 20px; border: solid 1px black">
The language plugin deals <b>only</b> with language specific details.
</blockquote>
</p>
<p>
The implication of this is that nearly all UI, and all "infrastructure"
code like listening on editor changes, scheduling parsing, implementing editor
actions etc. are handled by the GSF infrastructure. A language plugin should
just focus on specific features as they pertain to this language.
</p>
<p>
If you are trying to get started writing a new language using GSF,
see the <a href="getting-started.html">Getting Started</a> document,
which will try to order the tasks involved and point to the relevant
sections.
</p>
<h3>Example: Instant Rename</h3>
<p>
For an example of this, take a look at the following diagram. This shows
how "Instant Rename" works in GSF. The Instant Rename action itself
(editor action implementation, keybinding registration, etc.) is provided
by GSF. The language plugin implements two services:
<ul>
<li> A parser - which produces a parse tree. This is used for most
features in GSF.
<li> An "InstantRenamer" - a simple interface where the language plugin
is handed one of its own parse trees, and it needs to answer two
questions:
<ol>
<li>Is the given caret offset renameable? (And if not, provide
some error message to explain why, or an indication that refactoring
needs to be involved.</li>)
<li>Produce a set of offset ranges (begin to end) for all the
instances of this symbol that should be renamed together.
</li>
</ol>
</ul>
Here's the flow - the left side is what happens on the GSF side, and
the right side is what happens in the language plugin:
<br/>
<img src="instant-rename.png" />
<br/>
<br/>
One important thing to notice here is that on the GSF side, there
is <b>NO</b> interpretation of the parser result in any way. GSF asks
you to parse your code, and hands the parser result back to you
as part of the implementation of many features.
</p>
<p>
As you can see, GSF provides implementations for a lot of the UI
and interactions that you need for deep editing support for your language.
You implement your lexer, parser, and feature implementations like
code completion, go to declaration, quickfixes etc. based on analyzing
your own parse trees. You talk through feature specific APIs to GSF,
and GSF does the rest.
</p>
<h2>Architecture</h2>
<p>
Here's a rough architecture diagram for GSF:
<br/>
<img src="gsf-architecture.png" />
<br/>
<br/>
As you can see, language plugins talk mostly through the GSF APIs
to implement their features. GSF is in turn implemented in many cases
on top of existing NetBeans APIs. In some cases, the abstraction on
top of existing APIs (such as the Hyperlink API) is very thin. In other
cases, such as for quickfixes, mark occurrences or semantic highlighting,
GSF provides a lot of UI implementation under the hood on top of the
existing APIs. You might wonder why GSF should bother with features
such as code completion and hyperlinking when the existing APIs are
pretty complete. There are several reasons for this:
<ul>
<li>Simpler to learn from one place. The goal for GSF is to bring
language support of many languages in NetBeans up to the same depth
of support as we have for Java. (See
<a href="http://wiki.netbeans.org/EditorFeaturesChecklist">http://wiki.netbeans.org/EditorFeaturesChecklist</a>
for a document which tries to list all these features.)
If you have to go and hunt for various different NetBeans APIs
(navigation, hyperlinking, highlighting, etc.) it's a lot harder
to make sure you cover everything, and you in some cases have
to learn new concepts.
</li>
<li>
GSF can take advantage of synergy. For example, GSF knows how
to (a) parse your code, and (b) index your code, so it can hand
you an up to date parse tree when it's asking you to for example
compute the declaration location for an editor click. That makes
your job easier. As another example, because GSF implements both
your Hyperlink provider (Go To Declaration), and your code completion
documentation provider, it can register a tooltip which shows the
documentation for the symbol under the caret when you are
ctrl-hovering over that symbol. And so on.
</li>
<li>
GSF makes unit testing easier. This is another aspect of the synergy
point I brought up above. When you for example add a new
"semantic highlighting" implementation, writing a unit test for it
is as simple as writing a simple function call - pass in a source
file, and a caret location. The GSF unit testing infrastructure
will do the rest: It will parse your file, pass it to your semantic
highlighting feature implementation, produce a nice "golden file"
pretty print of your highlighting (see the unit testing section
of this document for details), and diff that result with the recorded
golden file (or create it if it doesn't exist). Thus, to unit test,
just create new testcases in the form of a source file, run the test
once to create a golden file, look at the golden file to make sure
it looks correct, and from now on you have a regression test which
makes sure that your semantic highlighter always produces the same
highlights for this source file.
</li>
<li>
GSF supports embedding. Thus, if you for example implement a GSF
hints provider for your language, when your language is embedded in
some other files, your quickfixes continue to work; they get run,
the error offsets from your AST get translated back to embedding
appropriate offsets, etc.
</li>
</ul>
The bigger point though is that for nearly all editing actions, there is
shared UI that should not be duplicated across modules. Things like
color and font definitions for syntax highlighting, error messages,
action names, keybindings, icons etc. should all be defined in one place
such that periodic maintenance keeps everything in sync, and such that
for example editor themes work across all intentional and unintentional
file types.
</p>
<p>
Furthermore, for many actions, there is a LOT of code behind the scenes
that is not language specific. Take mark occurrences for example.
The only language specific thing there is determining, for a given offset,
what the other corresponding occurrences are in terms of editor offsets.
Everything else: ensuring that the parse information is up to date,
listening on caret changes, painting highlights for the occurrences,
defining and implementing "goto previous occurrence" and "goto next occurrence",
and so on, should all be implemented in just one shared place.
</p>
<p>
One thing which is not obvious from the diagram above is that clients
are free to use NetBeans APIs directly. For example, in NetBeans 6.1,
the HTML module uses GSF for parsing, navigator, embedding etc. - but
it registers its own CompletionProvider using the NetBeans APIs
instead of GSF's code completion handler interface. It is possible
to combine GSF with for example Schliemann or a custom written
editor kit or custom written data loaders. More details about this
are found in the <a href="registration.html#UseCustomEditorKit">custom editor kit</a> section.
</p>
<h2>Lack of Extensibility</h2>
<p>
One of the key principles of GSF is to keep all UI in GSF. In the old days,
each language support would define all its own logical editor types, color
definitions, and so on. That made it really tricky to provide editor themes,
as well as keep things consistent. A literal string may have one color in one
file type, and another color in another filetype.
</p>
<p>
This has some implications for GSF:
<ul>
<li>
In the Java module, one of the key APIs you get is the ability
to schedule a user task: run the parse tree on a Java file,
and then run a task over the parse tree. (This is
<code>JavaSource.runUserActionTask()</code>).
This lets modules create arbitrary features for Java - and this
is how the various Java modules (navigation, hints, etc)
operate.
<br/><br/>
GSF does not push this approach (it's actually there, through
the <code>SourceModel</code> class, but you're discouraged
from using it).
<br/><br/>
This facility is typically used to implement specific features.
And the GSF Way is to implement the feature INSIDE GSF, and expose
a lean feature API to other language clients. If it's a feature
needed for language A, chances are it's needed for language B
as well, and putting user <b>feature</b> code in language plugins
is the opposite approach of what GSF is trying to do.
<br/><br/>
</li>
<li>
GSF tries to make it really easy to describe UI. It provides this
through the
<a href="org/netbeans/modules/gsf/api/ElementKind.html">ElementKind</a>
enumeration class. When creating code completion items, or navigation
items, just reference one of the constants there, such as
ElementKind.METHOD, to create an item which will look like a method,
consistent across languages. This will provide an icon, a color,
a code completion sorting priority etc. None of these are language
specific, which is why GSF wants to provide the definitions, and
language clients just reference them through the enumeration name.
<br/><br/>
This of course may seem like a straightjacket - the types of code
elements that GSF can address are hardcoded!!
<br/><br/>
Once again, the philosophy here is that GSF is trying to address
a wide swath of languages, but perhaps not every single one.
Rather than saying that say a Prolog language has a particular
code element that isn't provided by GSF today so Prolog should have
the option of adding it, I'd like to add new types to the ElementKind
enum. They can be added in every release without any backwards
compatibility problems. Chances are, if something is available in
one language, it's going to be needed by some other language as well.
Second, there's a pretty simple workaround - pick something SIMILAR.
In Ruby for example, I could have used the FIELD kind if there
wasn't an ATTRIBUTE kind.
<br/><br/>
Finally, there are backdoors. For example, the CompletionProposal
interface (implemented by code completion items) actually has
a <code>getIcon()</code> method, where you normally return null,
but you can override the icon there. This is how I return a
Ruby specific icon for Ruby keywords in code completion (and
ditto for JavaScript). But overriding UI is the exception, not
the norm.
<br/><br/>
</li>
<li>
GSF lets client format the data shown in code completion items;
there's the left hand side, the right hand side, as well as
different colors and attributes for parameter types, return types,
overridden methods, deprecated methods, etc. Rather than
letting clients format HTML directly (which would for example
cause color definitions to get hardcoded into the clients),
GSF provides an HTML formatter to clients, and when they need
to produce HTML, they use logical formatter methods to emit
HTML. The actual color definitions etc. are therefore provided
by GSF and clients are a bit more high level.
<br/><br/>
</li>
</ul>
</p>
<h2>Service Implementations</h2>
<p>
A language plugin basically registers a bunch of callbacks. These are invoked
at appropriate times by the infrastructure. There are two types of callbacks:
<ol>
<li>Service Implementations, such as lexing and parsing</li>
<li>Feature Implementations, such as code completion and quickfixes</li>
</ol>
Service Implementations basically implement basic services that are used
to drive other features. There service implementations are:
<ul>
<li>Lexing: Tokenize a document into syntactic elements</li>
<li>Parsing: Parse a document into semantic structure</li>
<li>Indexing: Given a parse result, store some information in a persistent
(and quickly searchable) index</li>
</ul>
The infrastructure will call your services at the right times. For example,
the lexer is asked to lex your document when it is opened in the editor.
It is also asked to update the token hierarchy immediately as the document
is edited. Similarly, the parser is called in a background thread shortly
after the document has been edited, or immediately when the result is needed
right away such as during code completion. And finally, the indexer is called
when you leave a file (to keep the index up to date), or at startup for
files that are new or have been updated outside the IDE.
</p>
<h3>Lexing</h3>
<p>
GSF uses the Lexer API in NetBeans directly. All GSF languages must provide
a Lexer language. It must also be registered in the Editors mime folder.
See the <a href="registration.html">Registration</a> section for more details.
</p>
<h3>Parsing</h3>
<p>
GSF provides a <a href="org/netbeans/modules/gsf/api/Parser.html">parser</a> interface.
You can register a parser, and GSF will call your parser when needed.
</p>
<blockquote style="background-color: #ffdddd; color: black; padding: 20px; border: solid 1px black">
NOTE: There is a new Parsing API underway. This will be something
equivalent to the Lexer API, but applied to parsing. The plan is for GSF
to remove its own Parser registration and interfaces and replace it with
the standard Parser API. This will allow embedding and coordination
not just among GSF-based languages, but for all languages that implement
the parsing API (such as Java, and C/C++, which are not GSF based).
</blockquote>
<p>
The Parser API basically just asks you to parse a given CharSequence,
and return the result as your own subclass of the
<a href="org/netbeans/modules/gsf/api/ParserResult.html">ParserResult</a> interface.
It is in your own parser result that you store your own AST reference.
You will probably need it to implement most of the features.
There is a separate document describing the <a href="parsing.html">parsing aspects</a>
of GSF. (Eventually you'll also want to implement
<a href="incremental-parsing.html">incremental parsing</a>.)
</p>
<h3>Indexing</h3>
<p>
Indexing lets you register a service which will extract information from your
parse results, and store them in an index which can be queried quickly later.
Your indexer will be called "at the right times" by the infrastructure:
<ol>
<li>At startup for all source files that are new or have changed since the
last IDE session
</li>
<li>After a file has been edited, and closed/left</li>
<li>Immediately if a file has been edited and somebody is trying to query the index
</ol>
This topic is described in a lot more detail in the separate
<a href="indexer.html">Indexing and Querying</a> document.
</p>
<h2>Feature Implementations</h2>
The service implementations above aren't really useful in their own right,
but they allow a number of features to be implemented. When implementing
additional feature interfaces, you get handed your token hierarchy from
your lexer, your parse tree and a handle to search the index populated
by your indexer, which you can use to respond to the feature requests.
<h3>Semantic Highlighting</h3>
<p>
</p>
<p>
To implement semantic highlighting, you need to
<a href="registration.html">register</a> an implementation of the
<a href="org/netbeans/modules/gsf/api/SemanticAnalyzer.html">SemanticAnalyzer</a> interface.
You will be given your own <code>ParserResult</code> object, and
you need to return a <code>Map<OffsetRange, Set<ColoringAttributes>></code>.
So, you'll iterate your own AST, look for things to highlight, such as
method definitions, parameters and unused variables, and for each node,
look up its source offsets (start, end integer offsets in the document),
and then place them in the document:
<pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px">
// Aha, this is an unused variable reference!
int nodeStart = node.getStartOffset();
int nodeEnd = node.getEndOffset();
result.put(new OffsetRange(nodeStart, nodeEnd), ColoringAttribute.UNUSED_SET);
</pre>
Mark Occurrences is very similar to this. You implement the
<a href="org/netbeans/modules/gsf/api/OccurrencesFinder.html">OccurrencesFinder</a> interface,
where you return a Set<OffsetRange> for the symbols that should be
highlighted as other occurrences of the symbol at the given offset.
</p>
<h3>Other Features</h3>
<p>
TODO: I plan to write some documents explaining how to do keystroke handling,
formatting, declaration finding, code completion, etc.
</p>
<a name="registration"/>
<h2>Registration</h2>
<p>
GSF services, such as the lexer, parser, and feature implementations,
must all be registered with the GSF infrastructure. For details on
how to do this, see the <a href="registration.html">Registration document</a>.
</p>
<h2>Embedding</h2>
<p>
GSF supports language embedding - supporting nested languages, like
JavaScript and CSS support inside HTML files, Ruby support (and JavaScript
and CSS support) inside ERb/RHTML files, and so on.
The mechanism of how this works is described fully in the
<a href="embedding.html">embedding document</a>.
</p>
<h2>GSF Diagnostic Tools</h2>
<p>
There is a separate module in the contrib repository, <code>gsf.tools</code>,
which provides a number of tools in the <b>Tools | GSF Development</b> menu.
Details are described in the <a href="gsf-tools.html">GSF Development Tools</a> document.
</p>
<a name="Classpath"/>
<h2>Classpath</h2>
GSF needs to integrated with the project system, in order to index source files at startup,
in order to support project-wide operations like Open Type, and so on. Details about
the project and classpath integration are discussed in more detail in the
<a href="classpath.html">Classpath document</a>.
<h2>Unit Testing</h2>
<p>
GSF makes it really easy to unit test your feature implementations.
Details on how to do this is described in the <a href="unit-testing.html">unit
testing document</a>.
</p>
<h2>Implementation Issues/Limitations</h2>
<p>
I will describe the various GSF limitations here. For now, they are
listed (in very brief form) in the following Wiki page:<br/>
<a href="http://wiki.netbeans.org/GsfIssues">http://wiki.netbeans.org/GsfIssues</a>
<br/>
</p>
<h2>TODO</h2>
<p>
<ul>
<li>
Describe the philosophy of the builtin ElementKind UI, icons, fonts etc. and
the general philosophy of pushing features INTO gsf rather than supporting
generic user task execution.
</li>
<li>
Describe the modules, imported and exported APIs, etc.
</li>
<li>
Document the GSF limitations
</li>
<li>Write documents for specific features, like code completion and keystroke handling</li>
</ul>
</p>
<br/>
<span style="color: #cccccc">Tor Norbye <tor@netbeans.org></span>
</body>
</html>
|
