How to draw thick and transparent insertion point in NSTextView

While I was working on drawing an insertion point in XVim project I had a really hard time to achieve my goal “drawing thick and transparent insertion point on a character”.

This does not look difficult since NSTextView has a method to override named “drawInsertionPointInRect:color:turnedOn:”. Actually just drawing “thick” insertion point is not difficult. But just overriding the method results in strange behaviour and not beautiful at all. Here I show what I tried and how I achieve the transparent thick insertion point.

If you are just interested in “how”, skip the part explaining what I tried and see the section “Solution” at the end of this post. There is a code for it. I tested the code in OS X 10.8 and also I must note that the code uses NSTextView’s internal method to override. This means that you may not use the code if you want your app to be in AppStore.

First attempt

As everyone can think of I first tried to subclass NSTextView and override “drawInsertionPointInRect:color:turnedOn:” (will be referred as “drawIPR” in this post). The code looks like following.

-(void)drawInsertionPointInRect:(NSRect)rect color:(NSColor)color turnedOn:(BOOL)flag {
    // Calculate the rect of a character under the insertion point
    NSRange charRange = NSMakeRange(self.selectedRange.location, 1);
    NSRect glyphRect = [[self layoutManager] boundingRectForGlyphRange:charRange inTextContainer:[self textContainer]];
    if( glyphRect.size.width == 0 ||
       [[NSCharacterSet newlineCharacterSet] characterIsMember:[[self string] characterAtIndex:self.selectedRange.location]]){
        // This handles when the insertion point is at the end of line where we cannot find any character to cover by a caret.
        // We keep the original width (1.0)
        glyphRect = rect;
    }
    [super drawInsertionPointInRect:glyphRect color:color turnedOn:flag];
}

This works somehow. But there are unacceptable problems.

No transparency. A caret completely fill the rect and we can not recognise the character under the caret.(Since the caret is blinking you still can see the character but not really beautiful)
When you move a cursor it draws thin(origina size) insertion point. You will see why this happens later.

Character ‘p’ is covered by a caret

OK. I have 2 problems. Solve them one by one.

Transparent color

So I tried to set the color’s alpha value. It’s like this.

-(void)drawInsertionPointInRect:(NSRect)rect color:(NSColor*)color turnedOn:(BOOL)flag {
    // Calculate the rect of a character under the insertion point
    NSRange charRange = NSMakeRange(self.selectedRange.location, 1);
    NSRect glyphRect = [[self layoutManager] boundingRectForGlyphRange:charRange inTextContainer:[self textContainer]];
    if( glyphRect.size.width == 0 ||
       [[NSCharacterSet newlineCharacterSet] characterIsMember:[[self string] characterAtIndex:self.selectedRange.location]]){
        // This handles when the insertion point is at the end of line where we cannot find any character to cover by a caret.
        // We keep the original width (1.0)
        glyphRect = rect;
    }
    // Set alpha of the color here.
    color = [color colorWithAlphaComponent:0.5]; // <--- newly added.
    [super drawInsertionPointInRect:glyphRect color:color turnedOn:flag];
}

Unfortunately this does not change anything. The reason is that inside the “drawIPR” in NSTextView it use NSRectFill to draw the insertion point and the alpha value of the color is completely ignored.

Drawing a caret by myself

Since the implementation of the “drawIRP” in NSTextView draws a caret by NSRectFill we can not use it to draw transparent caret. OK. We can re-implement the whole of the method. To re-implement it we have to know how the method works. It takes 3 arguments “rect”,”color” and “turnedOn”. the “rect” and “color” are apparent. The “turnedOn” argument specifies whether we should draw insertion point or clear the insertion point. The method is called with the “turnedOn” arugument set to YES and NO by turns, which results in blinking a caret. So the implementation should be like following.

-(void)drawInsertionPointInRect:(NSRect)rect color:(NSColor*)color turnedOn:(BOOL)flag {
    NSRange charRange = NSMakeRange(self.selectedRange.location, 1);
    NSRect glyphRect = [[self layoutManager] boundingRectForGlyphRange:charRange inTextContainer:[self textContainer]];
    if( glyphRect.size.width == 0 ||
       [[NSCharacterSet newlineCharacterSet] characterIsMember:[[self string] characterAtIndex:self.selectedRange.location]]){
        glyphRect = rect;
    }
    
    if( flag ){
        // Draw a caret
        color = [color colorWithAlphaComponent:0.5];
        [color set];
        NSRectFillUsingOperation(glyphRect,  NSCompositeSourceOver);
    }
    else{
        // Clear a caret
        // We have to redraw the character under the caret.
        // We should use "setNeedsDisplayInRect:" if possible. But to do so we have to keep the rect we drew somewhere.
        // This kind of optimisation is off topic of this post so just update the whole rect.
        [self setNeedsDisplay:YES];
    }
}

Yes, this solves the problem No.1 but not No.2. ( I think NSTextView’s implementation should be changed as this code works perfectly because this is the official way to override drawing insertion point).

The code above draws/clears a caret(with thick and transparency) correctly but when you move a cursor you see a caret without the thickness. Why? The reason why this happens is because NSTextView uses internal method to draw a caret and it is called when you move a cursor to redraw the caret immediately. The internal method is named “_drawInsertionPointInRect:color:”. Note that it starts with “_” and does not have “turenedOn” argument. (will be referred as “_drawIPR” in this post). The code in NSTextView should be like following according to my observation of the behavior. ( This is really simplified to make the point clear though. )

// This is guessed code of NSTextView implementation

// Private method to draw a caret.
// Note that this method only does "drawing". It does NOT have "turnedOn" flag.
-(void)_drawInsertionPointInRect:(NSRect)rect color:(NSColor*)color{
       [color set];
       NSRectFill(rect);
}

- (void)drawInsertionPointInRect:(NSRect)rect color:(NSColor*)color turnedOn:(BOOL)flag{
    if( flag ){
       // Call the internal method to draw insertion point 
       [_drawInsertionPointInRect:rect color:color]
    }else{
       [self setNeedsDisplayInRect:rect];
    }
}

// When you move a cursor, _drawRect (this is also an internal method) draws insertion point.
- (void)_drawRect:(NSRect)rect clip:(NSRect)clip{
    // At some point in this method...
    [self _drawInsertionPointInRect:NSMakeRect(x,y,1.0,height) color:_color];
}

The important points are

“_drawIPR” is called from multiple methods (not only from “drawIPR” but also from _drawRect:clip:)
In _drawRect:clip: it calls “_drawIPR” with hard coded width.

Overriding _drawInsertionPointInRect:color:

OK. So NSTextView uses “_drawIPR” to draw insertion point anyway. How about just overriding “_drawIPR” instead of “drawIPR”. The code should be like…

-(void)_drawInsertionPointInRect:(NSRect)rect color:(NSColor*)color{
    NSRange charRange = NSMakeRange(self.selectedRange.location, 1);
    NSRect glyphRect = [[self layoutManager] boundingRectForGlyphRange:charRange inTextContainer:[self textContainer]];
    if( glyphRect.size.width == 0 ||
       [[NSCharacterSet newlineCharacterSet] characterIsMember:[[self string] characterAtIndex:self.selectedRange.location]]){
        glyphRect = rect;
    }
    color = [color colorWithAlphaComponent:0.5];
    [color set];
    NSRectFillUsingOperation(glyphRect, NSCompositeSourceOver);
    // We do not call super calls since the method in the super class uses NSRectFill and calling it results in filling the rect with the color without transparency.
}

Note that we are not overriding “drawIPR”. This results in drawing a thick and transparent caret whenever NSTextView draws a caret. Unfortunately this leads another problem. The code is only “drawing” a caret but not “clearing” a caret to blink a caret. So once the caret is drawn the caret stays there and there is no blinking. Furthermore if you move a caret, all the characters the caret passes will be covered by the drawing and stays there.

Drawn carets are not cleared

The reason why the caret is not cleared correctly is because NSTextView does not know the rect which should be cleared. NSTextView tries to clear the area NSTextView draws a caret (which is a thin caret).

Clearing thick caret

Now the problem is clearing caret. The clearing a caret is done in “drawIPR” when the “turnedOn” argument is NO. We can tell NSTextView to redraw the view by calling “setNeedsDisplay” as we already saw.

Solution

The following code works well.

-(void)_drawInsertionPointInRect:(NSRect)rect color:(NSColor*)color{
    NSRange charRange = NSMakeRange(self.selectedRange.location, 1);
    NSRect glyphRect = [[self layoutManager] boundingRectForGlyphRange:charRange inTextContainer:[self textContainer]];
    if( glyphRect.size.width == 0 ||
       [[NSCharacterSet newlineCharacterSet] characterIsMember:[[self string] characterAtIndex:self.selectedRange.location]] ){
        glyphRect = rect;
    }
    color = [color colorWithAlphaComponent:0.5];
    [color set];
    NSRectFillUsingOperation(glyphRect, NSCompositeSourceOver);
    // We do not call super calls since the method in the super class uses NSRectFill and calling it results in filling the rect with the color without transparency.
}

- (void)drawInsertionPointInRect:(NSRect)rect color:(NSColor *)color turnedOn:(BOOL)flag{
    // Call super class first.
    [super drawInsertionPointInRect:rect color:color turnedOn:flag];
    // Then tell the view to redraw to clear a caret.
    if( !flag ){
        [self setNeedsDisplay:YES];
    }
}

The problem I can think of is that “setNeedsDisplay:YES” is not really optimised way to redraw the view since we are just interested in clearing the rect of a caret but not entire view. What we have to do for it is to keep the rect of a caret in a instance variable (by adding to the class) and call “setNeedsDisplayInRect:” to redraw that area.

(Appendix)Further investigation

I also did some further investigation for NSTextView internal.
The root of the problem of not clearing a caret is that NSTextView keeps the rect of a caret rect in an internal variable and use it to clear the caret. If we can update it by ourselves the problem should be solved. By using some Objective-C runtime functions I reverse engineered NSTextView class and I found the followings.

NSTextView has a private variable named “_ivars”
“_ivars” is a instence of NSTextViewIvars .
NSTextViewIvars has several variables (maybe 12 variables).
One of them is named “_insertionPointRect”
(I found “_insertionPointRectCache” too but do not know how this is used)

And also I found that after calling “_drawIPR” of NSTextView the value of the “_insertionPointRect” is updated.

Another solution (not recommended)

(If you are coming from top to get the code which works, use the code in “Solution” section.)
So what we should do is

Override “_drawIPR”
In “_drawIPR”, draw thick/transparent caret.
In “_drawIPR”, update “_insertionPointRect” to remember the rect of our caret.

So the code I got to achieve from this investigation is…

-(void)_drawInsertionPointInRect:(NSRect)rect color:(NSColor*)color{
    NSRange charRange = NSMakeRange(self.selectedRange.location, 1);
    NSRect glyphRect = [[self layoutManager] boundingRectForGlyphRange:charRange inTextContainer:[self textContainer]];
    if( glyphRect.size.width == 0 ||
       [[NSCharacterSet newlineCharacterSet] characterIsMember:[[self string] characterAtIndex:self.selectedRange.location]] ){
        glyphRect = rect;
    }
    color = [color colorWithAlphaComponent:0.5];
    [color set];
    NSRectFillUsingOperation(glyphRect, NSCompositeSourceOver);
    // Update internal variables (_insertionPointRect) in NSTextView
    id nsTextViewIvars;
    object_getInstanceVariable(self, "_ivars", (void**)&nsTextViewIvars);
    [nsTextViewIvars setValue:[NSValue valueWithRect:glyphRect] forKey:@"_insertionPointRect"];
}
// And you do not need to override "drawIPR" since NSTextView automatically clears a caret.

Yes. This clears a caret we draw correctly. But I sometimes see a caret draws several times at one time when you move a caret. Not too bad but not as beautiful as the code in “Solution” section.

Thank you for reading and I hope this helps you.