Hi

Here are the bits for wide character support in libcurses, libform,
libpanel, libmenu.

Apply the diff below and extract this tarball in src/lib (note this is
on a slow connection so please don't bother unless you are going to
test):

      http://nicm.ath.cx/~nicholas/ncursesw.tar.gz

It is a major bump for all four libs. The files are also on cvs in
~nicm/ncursesw/.

This builds all the libraries with the widechar code and also links them
to "w" versions (libncursesw, libformw, etc). Some OSs only put the
widechar stuff into libncursesw and not libncurses - I don't much like
knobs in the form of library names but if opinion is that we should go
that way then I can change it.

I've been running with this for a while on amd64 without seeing any
problems, and UTF-8 seems fine in mutt - the port picks it up fine, then
you just need to set LANG. It also seems fine on sparc64.

Any ports person like to volunteer to help test this/take care of
breakage? :-)

Tests, comments, oks?


Index: libform/Makefile
===================================================================
RCS file: /cvs/src/lib/libform/Makefile,v
retrieving revision 1.9
diff -u -p -r1.9 Makefile
--- libform/Makefile	12 Jan 2010 23:22:07 -0000	1.9
+++ libform/Makefile	19 Aug 2010 20:16:29 -0000
@@ -57,4 +57,16 @@ includes:
 	  ${INSTALL} ${INSTALL_COPY} -m 444 -o $(BINOWN) -g $(BINGRP) $$i \
 	  ${DESTDIR}/usr/include; done
 
+.include <bsd.own.mk>
+
+afterinstall:
+	-cd ${DESTDIR}${LIBDIR}; \
+	for i in ${_LIBS}; do \
+	    ln -f $$i `echo $$i | sed 's/form/formw/'`; \
+	done
+.if (${DEBUGLIBS:L} == "yes")
+	-cd ${DESTDIR}${LIBDIR}/debug; \
+	    ln -f lib${LIB}.a lib`echo ${LIB} | sed 's/form/formw/'`.a
+.endif
+
 .include <bsd.lib.mk>
Index: libform/shlib_version
===================================================================
RCS file: /cvs/src/lib/libform/shlib_version,v
retrieving revision 1.6
diff -u -p -r1.6 shlib_version
@@ -1,2 +1,2 @@
-major=4
+major=5
 minor=0
Index: ...
[ message continues ]

I'll run a bulk with it if noone beats me to it.

Landry

[ message continues ]

Here's a list of candidates that are likely to be affected:

audio/herrie.log:cc -c ./src/audio_file.c -O2 -pipe  -I/usr/local/include -DAPP_NAME=\"herrie\"      -DAPP_VERSION=\"2.2\"     -I/usr/local/include -I/usr/local/include/ncursesw     -I/usr/local/include/ncurses -DCURSES_HEADER=\<ncurses.h\> -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include -pthread -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include -DBUILD_HTTP -DBUILD_SCROBBLER -I/usr/local/include -DBUILD_MP3 -DBUILD_RES_INIT -DBUILD_SNDFILE -DBUILD_NLS -DBUILD_VORBIS -DBUILD_AO  -o audio_file.o
audio/ncmpc.log:checking for initscr in -lncursesw... no
audio/pms.log:checking for working ncursesw... no
comms/c3270.log:checking for newterm in -lncursesw... no
databases/mysql.log:checking for tgetent in -lncursesw... no
devel/tig.log:checking for ncursesw/ncurses.h... no
editors/zile.log:checking for working ncursesw... no
games/clines.log:checking for initscr in -lncursesw... no
graphics/libcaca.log:checking for initscr in -lncursesw... no
lang/ghc.log:checking for setupterm in -lncursesw... no
lang/swi-prolog.log:checking for main in -lncursesw... no
mail/abook.log:checking for initscr in -lncursesw... no
mail/mutt/snapshot.log:checking for waddnwstr in -lncursesw... no
mail/mutt/stable.log:checking for waddnwstr in -lncursesw... no
misc/dialog.log:checking if you want the wide-curses features... no
misc/lifelines.log:checking for tparm in -lncursesw... no
net/mcabber.log:checking for waddnwstr in -lncursesw... no
net/pidgin.log:checking for initscr in -lncursesw... no
print/texlive/base.log:checking if you want the wide-curses features... no
productivity/calcurse.log:checking for initscr in -lncursesw... no
shells/zsh.log:checking for ncursesw/ncurses.h... no
sysutils/ncdu.log:checking for initscr in -lncursesw... no
sysutils/testdisk.log:checking for initscr in -lncursesw... no
sysutils/vifm.log:checking for initscr in -lncursesw... no
textproc/aspell/core.log:checking for wide character ...
[ message continues ]

Fails with ../../src/hdrs/mycurses.h:8:29: error: ncursesw/curses.h: No

fails because it looks for a 'get_wch' #define in ncurses.h

Fails at

Other ports package fine... some will probably need to be checked for
WANTLIB (ncurses -> ncursesw). Some of the configure script seem to
only check for ncursesw/ncurses.h header, and don't bother trying
-lncursesw.

cvs:~landry/ncursesw-found.gz contains the list of 'ncursesw' occurences in
all the build logs for the bulk.

Landry

[ message continues ]

Hi


I don't think we're going to install this so this needs to be fixed in

Hmm. get_wch is a macro but it should be there. Does it define
_XOPEN_SOURCE_EXTENDED?


Again seems like it is probably missing _XOPEN_SOURCE_EXTENDED.

Maybe we should remove the need for that define since we have ncursesw
by default.


[ message continues ]

The configure test run is more or less:

#define _XOPEN_SOURCE_EXTENDED
#include </usr/include/ncurses.h>

int
main ()
{
#ifndef get_wch
# error get_wch not found!
#endif
;
return 0;
}

conftest.c:126:8: error: #error get_wch not found!


It is defined in Src/system.h

#ifndef ZSH_NO_XOPEN
# ifdef ZSH_CURSES_SOURCE
#  define _XOPEN_SOURCE_EXTENDED 1
# else
#  ifdef MULTIBYTE_SUPPORT
/*
 * Needed for wcwidth() which is part of XSI.
 * Various other uses of the interface mean we can't get away with just
 * _XOPEN_SOURCE.
 */
#   define _XOPEN_SOURCE_EXTENDED 1
#  endif /* MULTIBYTE_SUPPORT */
# endif /* ZSH_CURSES_SOURCE */
#endif /* ZSH_NO_XOPEN */

but ZSH_NO_XOPEN is defined to 1... given by this configure.ac snippet

[[case "$host_os" in
  *openbsd*|*freebsd5*|*freebsd6.[012]*|*aix*)
  zsh_cv_no_xopen=yes

Landry

[ message continues ]

Odd, because if I run this test it is fine. Maybe it is defining
NCURSES_NOMACROS somewhere, although that would seem silly if it is


So it probably isn't defining it, but is then using cchar_t (under
HAVE_SETCCHAR which must be detected a different way since it is under

[ message continues ]

config.log should tell you more..

[ message continues ]

Hmm. I can't reproduce this on amd64, make configure passes:

checking for initscr in -lncursesw... yes
checking for update_panels in -lpanelw... yes
checking /usr/include/ncursesw/ncurses.h usability... no
checking /usr/include/ncursesw/ncurses.h presence... no
checking for /usr/include/ncursesw/ncurses.h... no
checking /usr/include/ncurses.h usability... yes
checking /usr/include/ncurses.h presence... yes
checking for /usr/include/ncurses.h... yes
checking if /usr/include/ncurses.h supports wide characters... yes

And the build fails later instead looking for cchar_t:

 cc -DHAVE_CONFIG_H -I. -I/usr/ports/pobj/pidgin-2.7.3/pidgin-2.7.3/finch/libgnt -I../.. -pthread -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include -I/usr/include/ -Wall -DPURPLE_DISABLE_DEPRECATED -DPIDGIN_DISABLE_DEPRECATED -DFINCH_DISABLE_DEPRECATED -DGNT_DISABLE_DEPRECATED -Waggregate-return -Wcast-align -Wdeclaration-after-statement -Wendif-labels -Werror-implicit-function-declaration -Wextra -Wno-sign-compare -Wno-unused-parameter -Wformat-security -Winit-self -Wmissing-declarations -Wmissing-noreturn -Wmissing-prototypes -Wpointer-arith -Wundef -I/usr/local/include/libxml2 -I/usr/local/include -I/usr/local/include/python2.5 -I/usr/local/include -I/usr/include/ncursesw -O2 -pipe -MT gntwm.lo -MD -MP -MF .deps/gntwm.Tpo -c /usr/ports/pobj/pidgin-2.7.3/pidgin-2.7.3/finch/libgnt/gntwm.c  -fPIC -DPIC -o .libs/gntwm.o
/usr/ports/pobj/pidgin-2.7.3/pidgin-2.7.3/finch/libgnt/gntwm.c: In function 'work_around_for_ncurses_bug':
/usr/ports/pobj/pidgin-2.7.3/pidgin-2.7.3/finch/libgnt/gntwm.c:169: error: 'cchar_t' undeclared (first use in this function)
/usr/ports/pobj/pidgin-2.7.3/pidgin-2.7.3/finch/libgnt/gntwm.c:169: error: (Each undeclared identifier is reported only once
/usr/ports/pobj/pidgin-2.7.3/pidgin-2.7.3/finch/libgnt/gntwm.c:169: error: for each function it appears in.)
/usr/ports/pobj/pidgin-2.7.3/pidgin-2.7.3/finch/libgnt/gntwm.c:169: error: expected ';' before 'ch'
...

For which this might be ...
[ message continues ]

I think the problem I'm seeing is just a pidgin bug - python does define
_XOPEN_SOURCE_EXTENDED on Linux but it doesn't seem that it needs to on
OpenBSD so I don't think pidgin should rely on it.

FreeBSD/NetBSD ports don't build the console UI for pidgin so possibly
it just hasn't been tested with ncursesw.

I still can't reproduce your problem Landry, don't know why :-/.



[ message continues ]

Along with this, we also want to enable support in lynx.

(Display looks fine, but the cursor position is wrong when you enter
non-ASCII text into an input field.)

Index: gnu/usr.bin/lynx/Makefile.bsd-wrapper
===================================================================
RCS file: /cvs/src/gnu/usr.bin/lynx/Makefile.bsd-wrapper,v
retrieving revision 1.18
diff -u -p -r1.18 Makefile.bsd-wrapper
--- gnu/usr.bin/lynx/Makefile.bsd-wrapper	8 Aug 2009 12:20:30 -0000	1.18
+++ gnu/usr.bin/lynx/Makefile.bsd-wrapper	24 Aug 2010 14:07:16 -0000
@@ -11,7 +11,7 @@ OPT=-O1
 .endif
 
 GNUCFLAGS=	CC="${CC}" CFLAGS="${CFLAGS} ${COPTS} ${OPT}"
-CONFIGURE_ARGS= --with-ssl=/usr --enable-ipv6 --enable-debug
+CONFIGURE_ARGS= --with-ssl=/usr --enable-widec --enable-ipv6 --enable-debug
 HTMLDIR=	/usr/share/doc/html
 HELPFILES=	keystrokes/alt_edit_help.html keystrokes/bookmark_help.html \
 		keystrokes/cookie_help.html keystrokes/dired_help.html \
-- 
Christian "naddy" Weisgerber                          naddy@mips.inka.de

[ message continues ]

Thanks, I had lynx on my list to look at but hadn't got to it yet :-)



[ message continues ]

And now avec widechar man pages and MLINKs (no other changes).


Index: libform/Makefile
===================================================================
RCS file: /cvs/src/lib/libform/Makefile,v
retrieving revision 1.9
diff -u -p -r1.9 Makefile
--- libform/Makefile	12 Jan 2010 23:22:07 -0000	1.9
+++ libform/Makefile	25 Aug 2010 08:31:31 -0000
@@ -57,4 +57,16 @@ includes:
 	  ${INSTALL} ${INSTALL_COPY} -m 444 -o $(BINOWN) -g $(BINGRP) $$i \
 	  ${DESTDIR}/usr/include; done
 
+.include <bsd.own.mk>
+
+afterinstall:
+	-cd ${DESTDIR}${LIBDIR}; \
+	for i in ${_LIBS}; do \
+	    ln -f $$i `echo $$i | sed 's/form/formw/'`; \
+	done
+.if (${DEBUGLIBS:L} == "yes")
+	-cd ${DESTDIR}${LIBDIR}/debug; \
+	    ln -f lib${LIB}.a lib`echo ${LIB} | sed 's/form/formw/'`.a
+.endif
+
 .include <bsd.lib.mk>
Index: libform/shlib_version
===================================================================
RCS file: /cvs/src/lib/libform/shlib_version,v
retrieving revision 1.6
diff -u -p -r1.6 shlib_version
@@ -1,2 +1,2 @@
-major=4
+major=5
 minor=0
Index: libpanel/Makefile
===================================================================
RCS file: /cvs/src/lib/libpanel/Makefile,v
retrieving revision 1.8
diff -u -p -r1.8 Makefile
--- libpanel/Makefile	12 Jan 2010 23:22:08 -0000	1.8
+++ libpanel/Makefile	25 Aug 2010 08:31:31 -0000
@@ -20,4 +20,16 @@ includes:
 	  ${INSTALL} ${INSTALL_COPY} -m 444 -o $(BINOWN) -g $(BINGRP) $$i \
 	  ${DESTDIR}/usr/include; done
 
+.include <bsd.own.mk>
+
+afterinstall:
+	-cd ${DESTDIR}${LIBDIR}; \
+	for i in ${_LIBS}; do \
+	    ln -f $$i `echo $$i | sed 's/panel/panelw/'`; \
+	done
+.if (${DEBUGLIBS:L} == "yes")
+	-cd ${DESTDIR}${LIBDIR}/debug; \
+	    ln -f lib${LIB}.a lib`echo ${LIB} | sed 's/panel/panelw/'`.a
+.endif
+
 .include <bsd.lib.mk>
Index: libpanel/shlib_version
===================================================================
RCS file: /cvs/src/lib/libpanel/shlib_version,v
retrieving revision 1.7
diff -u -p -r1.7 ...
[ message continues ]

Hi,


I've tested mutt, tig and ncmpc on amd64, and they all work fine
in a standard xterm. However, when I use any of the three apps in
a tmux session (tmux -u) I get some strange visuals when a special
character is displayed (until the window is totally redrawn, when
I change its size for example). Here is a screen of a nested mutt
(left) and the same directly opened in an xterm (right):
	http://nolizard.org/obsd/tmux_ncurses_utf8.jpg


Below is the modification to tig's configure in order to check
both ncursesw/ncurses.h and ncurses.h for widechar.

Martin

--- configure.orig	Sat Jun 12 21:08:59 2010
+++ configure	Wed Aug 25 17:27:28 2010
@@ -3201,16 +3220,21 @@
 done


-for ac_header in ncursesw/ncurses.h
+for ac_header in ncursesw/ncurses.h ncurses.h
 do :
-  ac_fn_c_check_header_mongrel "$LINENO" "ncursesw/ncurses.h" "ac_cv_header_ncursesw_ncurses_h" "$ac_includes_default"
-if test "x$ac_cv_header_ncursesw_ncurses_h" = x""yes; then :
+  as_ac_Header=`$as_echo "ac_cv_header_$ac_header
+" | $as_tr_sh`
+ac_fn_c_check_header_mongrel "$LINENO" "$ac_header" "$as_ac_Header" "$ac_includes_default"
+eval as_val=\$$as_ac_Header
+   if test "x$as_val" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
-#define HAVE_NCURSESW_NCURSES_H 1
+#define `$as_echo "HAVE_$ac_header
+" | $as_tr_cpp` 1
+
 _ACEOF
  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing initscr" >&5
 $as_echo_n "checking for library containing initscr... " >&6; }
-if test "${ac_cv_search_initscr+set}" = set; then :
+if test "${ac_cv_search_initscr_+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_func_search_save_LIBS=$LIBS

[ message continues ]

Show me the output from "tmux lsc" and "tmux showw -g|grep utf8" with
the client attached.




[ message continues ]

aer@goiaba $ tmux lsc
/dev/ttyp1: 0 [80x53 xterm-256color] (utf8)
aer@goiaba $ tmux showw -g|grep utf8
utf8 on

[ message continues ]

Hmm. I can't reproduce this.

What arch are you using? -current?

I do, in uxterm:

$ tmux kill-server
$ export LANG=en_US.UTF-8
$ tmux new
$ echo $LANG
en_US.UTF-8
$ mutt

And open a UTF-8 email, which appears fine, including wide characters.

Can you give me the exact steps you are using?



[ message continues ]

Yeah, I found the problem, I was reseting the $TERM variable in my kshrc.
Everything work like a charm now. Thank you for your work!

Martin

[ message continues ]

I have fixes for the zsh, lifelines ports and the problem I can
reproduce with pidgin, which I can send to maintainers.

Can anyone reproduce Landry's build issue with pidgin? I think it is a
problem with pidgin not ncursesw.

Any other problems?

So shall this go in? Any oks?

Cheers



[ message continues ]